[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 506241912 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2608MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002370] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004012] kvm-guest: setup PV IPIs [ 0.007250] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009031] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.011137] LSM: Security Framework initializing [ 0.013042] Yama: becoming mindful. [ 0.013761] SELinux: Initializing. [ 0.014074] *** VALIDATE selinux *** [ 0.022236] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026535] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028154] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029115] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030109] *** VALIDATE tmpfs *** [ 0.031403] *** VALIDATE proc *** [ 0.032221] *** VALIDATE cgroup *** [ 0.033005] *** VALIDATE cgroup2 *** [ 0.034224] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035151] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037025] Spectre V2 : User space: Vulnerable [ 0.038005] Speculative Store Bypass: Vulnerable [ 0.041110] debug: unmapping init [mem 0xffffffff84859000-0xffffffff84860fff] [ 0.043174] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044700] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045028] ... version: 2 [ 0.046014] ... bit width: 48 [ 0.047014] ... generic registers: 4 [ 0.048000] ... value mask: 0000ffffffffffff [ 0.048018] ... max period: 00007fffffffffff [ 0.049011] ... fixed-purpose events: 3 [ 0.050011] ... event mask: 000000070000000f [ 0.051321] rcu: Hierarchical SRCU implementation. [ 0.053496] smp: Bringing up secondary CPUs ... [ 0.054580] x86: Booting SMP configuration: [ 0.055027] .... node #0, CPUs: #1 #2 #3 [ 0.062243] smp: Brought up 1 node, 4 CPUs [ 0.064013] smpboot: Max logical packages: 1 [ 0.065012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.110192] node 0 deferred pages initialised in 43ms [ 0.113389] devtmpfs: initialized [ 0.114200] x86/mm: Memory block size: 128MB [ 0.116903] gcov: version magic: 0x41383552 [ 0.118325] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.119074] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.120247] pinctrl core: initialized pinctrl subsystem [ 0.121170] [ 0.121537] ************************************************************* [ 0.122013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.123008] ** ** [ 0.124011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.125011] ** ** [ 0.126008] ** This means that this kernel is built to expose internal ** [ 0.127014] ** IOMMU data structures, which may compromise security on ** [ 0.128010] ** your system. ** [ 0.129011] ** ** [ 0.130011] ** If you see this message and you are not debugging the ** [ 0.131011] ** kernel, report this immediately to your vendor! ** [ 0.132009] ** ** [ 0.133010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.134011] ************************************************************* [ 0.135614] NET: Registered protocol family 16 [ 0.136424] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.137053] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.138051] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.140127] cpuidle: using governor menu [ 0.142970] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.145484] PCI: Using configuration type 1 for base access [ 0.148142] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.157061] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.159021] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.162649] cryptd: max_cpu_qlen set to 1000 [ 0.165294] ACPI: Added _OSI(Module Device) [ 0.167013] ACPI: Added _OSI(Processor Device) [ 0.169014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.170012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.175230] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.182453] ACPI: Interpreter enabled [ 0.184063] ACPI: PM: (supports S0 S3 S4 S5) [ 0.185013] ACPI: Using IOAPIC for interrupt routing [ 0.187104] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.191459] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.200811] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.203046] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.206018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.209077] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.214447] acpiphp: Slot [2] registered [ 0.216106] acpiphp: Slot [5] registered [ 0.218163] acpiphp: Slot [6] registered [ 0.219117] acpiphp: Slot [7] registered [ 0.221103] acpiphp: Slot [8] registered [ 0.222120] acpiphp: Slot [9] registered [ 0.224112] acpiphp: Slot [10] registered [ 0.225000] acpiphp: Slot [3] registered [ 0.225000] acpiphp: Slot [4] registered [ 0.227124] acpiphp: Slot [11] registered [ 0.229099] acpiphp: Slot [12] registered [ 0.230105] acpiphp: Slot [13] registered [ 0.232090] acpiphp: Slot [14] registered [ 0.233092] acpiphp: Slot [15] registered [ 0.236117] acpiphp: Slot [16] registered [ 0.237128] acpiphp: Slot [17] registered [ 0.239125] acpiphp: Slot [18] registered [ 0.241108] acpiphp: Slot [19] registered [ 0.242098] acpiphp: Slot [20] registered [ 0.244098] acpiphp: Slot [21] registered [ 0.245101] acpiphp: Slot [22] registered [ 0.247113] acpiphp: Slot [23] registered [ 0.248112] acpiphp: Slot [24] registered [ 0.250105] acpiphp: Slot [25] registered [ 0.252098] acpiphp: Slot [26] registered [ 0.253118] acpiphp: Slot [27] registered [ 0.255142] acpiphp: Slot [28] registered [ 0.257121] acpiphp: Slot [29] registered [ 0.258122] acpiphp: Slot [30] registered [ 0.260174] acpiphp: Slot [31] registered [ 0.261088] PCI host bridge to bus 0000:00 [ 0.263020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.265027] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.267023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.270025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.273024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.276027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.278195] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.281355] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.285845] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.297020] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.303058] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.306016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.308018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.311021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.313505] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.315927] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.319047] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.321759] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.328027] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.340018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.345015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.350994] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.367015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.376018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.405088] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.417943] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.426018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.432016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.451019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.463755] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.474017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.485022] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.502018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.515347] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.523018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.537017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.562024] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.573961] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.579016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.586017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.601020] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.612328] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.619018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.626018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.644016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.653360] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.656881] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.659595] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.661243] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.663171] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.667133] iommu: Default domain type: Passthrough [ 0.668371] SCSI subsystem initialized [ 0.669193] ACPI: bus type USB registered [ 0.669899] usbcore: registered new interface driver usbfs [ 0.671067] usbcore: registered new interface driver hub [ 0.671985] usbcore: registered new device driver usb [ 0.673148] pps_core: LinuxPPS API ver. 1 registered [ 0.673901] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.675064] PTP clock support registered [ 0.676115] EDAC MC: Ver: 3.0.0 [ 0.677104] PCI: Using ACPI for IRQ routing [ 0.678650] NetLabel: Initializing [ 0.679009] NetLabel: domain hash size = 128 [ 0.679705] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.681072] NetLabel: unlabeled traffic allowed by default [ 0.682141] vgaarb: loaded [ 0.684242] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.685010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.692479] clocksource: Switched to clocksource kvm-clock [ 0.797391] VFS: Disk quotas dquot_6.6.0 [ 0.798959] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.801220] *** VALIDATE ramfs *** [ 0.802325] *** VALIDATE hugetlbfs *** [ 0.803798] pnp: PnP ACPI init [ 0.806146] pnp: PnP ACPI: found 6 devices [ 0.829420] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.832537] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.834557] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.836361] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.838183] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.840164] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.842664] NET: Registered protocol family 2 [ 0.845040] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.849836] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.853504] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.858612] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.863907] TCP: Hash tables configured (established 65536 bind 65536) [ 0.866542] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.870085] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.873506] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.877266] NET: Registered protocol family 1 [ 0.881226] RPC: Registered named UNIX socket transport module. [ 0.883457] RPC: Registered udp transport module. [ 0.885410] RPC: Registered tcp transport module. [ 0.887339] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.889562] NET: Registered protocol family 44 [ 0.892428] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.894498] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.897553] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.900135] PCI: CLS 0 bytes, default 64 [ 0.901967] Unpacking initramfs... [ 2.303663] debug: unmapping init [mem 0xffff9681bcc54000-0xffff9681bffbffff] [ 2.307779] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.310156] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 2.313704] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.797132] Initialise system trusted keyrings [ 2.798764] Key type blacklist registered [ 2.800695] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.807852] zbud: loaded [ 2.810326] *** VALIDATE nfs *** [ 2.811543] *** VALIDATE nfs4 *** [ 2.813109] pstore: using deflate compression [ 2.816464] Platform Keyring initialized [ 2.906216] NET: Registered protocol family 38 [ 2.907538] Key type asymmetric registered [ 2.908780] Asymmetric key parser 'x509' registered [ 2.909956] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.911850] io scheduler mq-deadline registered [ 2.912873] io scheduler kyber registered [ 2.913938] io scheduler bfq registered [ 2.915252] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.917595] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.920089] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.921774] ACPI: Power Button [PWRF] [ 2.925092] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.930458] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.940294] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.945212] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.957842] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.983658] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.009848] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.015780] Non-volatile memory driver v1.3 [ 3.016941] Linux agpgart interface v0.103 [ 3.040307] virtio_blk virtio1: [vda] 146712 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.042188] vda: detected capacity change from 0 to 75116544 [ 3.054157] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.056348] vdb: detected capacity change from 0 to 1073741824 [ 3.072774] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.074968] vdc: detected capacity change from 0 to 2621440000 [ 3.088222] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.090312] vdd: detected capacity change from 0 to 2621440000 [ 3.103547] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.105480] vde: detected capacity change from 0 to 4294967296 [ 3.117333] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.119289] vdf: detected capacity change from 0 to 4294967296 [ 3.123988] libphy: Fixed MDIO Bus: probed [ 3.133188] usbcore: registered new interface driver usbserial_generic [ 3.135248] usbserial: USB Serial support registered for generic [ 3.137145] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.141471] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.143361] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.145605] mousedev: PS/2 mouse device common for all mice [ 3.148645] rtc_cmos 00:05: RTC can wake from S4 [ 3.152776] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.155219] rtc_cmos 00:05: registered as rtc0 [ 3.159829] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.160837] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.164877] intel_pstate: CPU model not supported [ 3.171139] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.175241] hid: raw HID events driver (C) Jiri Kosina [ 3.177150] usbcore: registered new interface driver usbhid [ 3.179172] usbhid: USB HID core driver [ 3.180431] drop_monitor: Initializing network drop monitor service [ 3.182571] Initializing XFRM netlink socket [ 3.184084] NET: Registered protocol family 10 [ 3.187313] Segment Routing with IPv6 [ 3.188193] NET: Registered protocol family 17 [ 3.189454] mpls_gso: MPLS GSO support [ 3.195148] RAS: Correctable Errors collector initialized. [ 3.196395] AVX version of gcm_enc/dec engaged. [ 3.197310] AES CTR mode by8 optimization enabled [ 3.259693] sched_clock: Marking stable (3259661104, 0)->(4176457119, -916796015) [ 3.262438] registered taskstats version 1 [ 3.264723] Loading compiled-in X.509 certificates [ 3.266168] zswap: loaded using pool lzo/zbud [ 3.288302] Key type big_key registered [ 3.300780] Key type encrypted registered [ 3.301967] ima: No TPM chip found, activating TPM-bypass! [ 3.303390] ima: Allocated hash algorithm: sha1 [ 3.304594] ima: No architecture policies found [ 3.305870] evm: Initialising EVM extended attributes: [ 3.307040] evm: security.selinux [ 3.307961] evm: security.ima [ 3.309328] evm: security.capability [ 3.310620] evm: HMAC attrs: 0x1 [ 3.312970] rtc_cmos 00:05: setting system clock to 2026-09-08 08:56:15 UTC (1788857775) [ 3.319909] debug: unmapping init [mem 0xffffffff85803000-0xffffffff859fffff] [ 3.322949] debug: unmapping init [mem 0xffffffff84582000-0xffffffff84858fff] [ 3.331106] Write protecting the kernel read-only data: 28672k [ 3.333604] debug: unmapping init [mem 0xffffffff82c03000-0xffffffff82dfffff] [ 3.335557] debug: unmapping init [mem 0xffffffff83514000-0xffffffff835fffff] [ 3.363929] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.369437] systemd[1]: Detected virtualization kvm. [ 3.371073] systemd[1]: Detected architecture x86-64. [ 3.372440] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.397656] systemd[1]: No hostname configured. [ 3.398773] systemd[1]: Set hostname to . [ 3.400280] random: systemd: uninitialized urandom read (16 bytes read) [ 3.401727] systemd[1]: Initializing machine ID from random generator. [ 3.441927] random: ln: uninitialized urandom read (6 bytes read) [ 3.551844] random: systemd: uninitialized urandom read (16 bytes read) [ 3.555272] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.559934] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.564147] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Control Socket. Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Swap. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target Timers. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.158872] device-mapper: uevent: version 1.0.3 [ 4.161160] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ 4.655321] random: fast init done [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.837964] virtio_net virtio0 ens2: renamed from eth0 [ 4.900746] scsi host0: ata_piix [ 4.907529] scsi host1: ata_piix [ 4.933565] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.936374] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.560390] dracut-initqueue[582]: RTNETLINK answers: File exists [ 9.820219] random: crng init done [ 9.821539] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.110581] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.155483] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.398261] SELinux: Disabled at runtime. [ 11.458456] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.465250] systemd[1]: Detected virtualization kvm. [ 11.467043] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.927337] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.929831] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.934375] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.937526] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.939400] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.948991] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.961001] systemd[1]: Mounting POSIX Message Queue File System... Mounting POSIX Message Queue File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... Starting Remount Root and Kernel File Systems... Mounting Huge Pages File System... Activating swap /dev/disk/by-label/SWAP... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Mounting Kernel Debug File System... [ OK ] Reached target RPC Port Mapper. [ 12.066233] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.425648] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.751338] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.783338] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.872838] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.895894] EDAC sbridge: Ver: 1.1.2 [ 14.077815] Key type dns_resolver registered [ 14.375507] NFS: Registering the id_resolver key type [ 14.378371] Key type id_resolver registered [ 14.380592] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg249-server login: [ 40.884870] libcfs: loading out-of-tree module taints kernel. [ 40.944774] Key type ._llcrypt registered [ 40.955310] Key type .llcrypt registered [ 41.140073] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 62.940640] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 64.946205] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 64.970433] alg: No test for adler32 (adler32-zlib) [ 66.535923] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 67.644967] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [ 69.522347] Key type lgssc registered [ 71.831416] Lustre: Echo OBD driver; http://www.lustre.org/ [ 79.864100] hrtimer: interrupt took 2156159 ns [ 92.198533] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 142.009300] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 157.006740] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 157.047597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 158.463842] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 158.537558] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 158.757946] Lustre: lustre-MDT0000: new disk, initializing [ 158.985885] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 159.041541] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 164.548268] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 180.986685] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 181.101517] Lustre: 6511:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 181.144575] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 181.150108] Lustre: Skipped 1 previous similar message [ 181.254889] Lustre: lustre-MDT0001: new disk, initializing [ 181.333121] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 181.373987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 181.384481] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 186.742062] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 191.859814] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 201.829345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 202.207106] Lustre: lustre-OST0000: new disk, initializing [ 202.215375] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 202.225732] Lustre: 8450:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 202.300094] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 202.799114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 202.809668] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 202.861288] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 210.070764] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 225.753688] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 225.961536] Lustre: lustre-OST0001: new disk, initializing [ 225.966874] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 225.982388] Lustre: 9522:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 226.090172] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 232.538558] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 232.550122] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 232.615810] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 233.590570] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 245.844354] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 256.826965] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 265.721597] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing check_logdir /tmp/testlogs/ [ 271.925558] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing yml_node [ 276.083090] Lustre: DEBUG MARKER: Client: 2.17.57.111 [ 279.180312] Lustre: DEBUG MARKER: MDS: 2.17.57.111 [ 282.359551] Lustre: DEBUG MARKER: OSS: 2.17.57.111 [ 284.273872] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 8 05:00:55 EDT 2026 [ 305.812352] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 307.689390] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 329.697268] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 329.710566] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 329.745421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 330.216027] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 330.246639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 330.253074] Lustre: Skipped 3 previous similar messages [ 331.727928] Lustre: server umount lustre-MDT0000 complete [ 336.075131] LustreError: 6503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788858108 with bad export cookie 364388913159387439 [ 336.077164] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 336.094091] LustreError: 6503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 340.448887] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 340.450773] LustreError: 6518:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 340.450972] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 340.467183] Lustre: Skipped 3 previous similar messages [ 340.491357] LustreError: 6518:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 341.473165] LustreError: 8458:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 341.498995] LustreError: 8458:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 342.560794] Lustre: server umount lustre-MDT0001 complete [ 347.367569] Lustre: server umount lustre-OST0000 complete [ 351.509661] Lustre: server umount lustre-OST0001 complete [ 359.293739] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 370.927440] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 414.573991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 415.029630] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 415.081766] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 415.184245] Lustre: lustre-MDT0000: new disk, initializing [ 415.324339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 415.374619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 421.673104] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 432.861920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 433.030474] Lustre: 15089:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 433.093485] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 433.097611] Lustre: Skipped 1 previous similar message [ 433.235699] Lustre: lustre-MDT0001: new disk, initializing [ 433.435643] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 433.546878] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 433.567897] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 439.158883] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 449.062384] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 455.461862] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 463.310413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 463.638963] Lustre: lustre-OST0000: new disk, initializing [ 463.643230] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 463.651951] Lustre: 17062:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 463.732445] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 465.061931] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 465.075062] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 465.222061] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 470.391636] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 482.232991] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 489.012867] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 489.382778] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 496.985734] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 497.411428] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 501.220385] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 501.224156] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 501.229379] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 501.229387] Lustre: Skipped 3 previous similar messages [ 501.230448] Lustre: Skipped 1 previous similar message [ 501.262179] LustreError: Skipped 1 previous similar message [ 505.511277] Lustre: server umount lustre-OST0000 complete [ 510.433430] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 510.447100] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 510.468167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 510.470880] Lustre: Skipped 1 previous similar message [ 514.882829] Lustre: server umount lustre-MDT0000 complete [ 516.587301] LustreError: 15097:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 516.613904] LustreError: 15097:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 519.486187] LustreError: 15885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788858291 with bad export cookie 364388913159388972 [ 519.491956] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 519.496543] LustreError: 15885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 519.780779] Lustre: server umount lustre-MDT0001 complete [ 526.068562] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 05:04:57 (1788858297) [ 547.124544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 547.633883] LustreError: 19936:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 547.723368] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 552.935087] LustreError: 19937:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 553.036764] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 561.250611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 566.868962] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 575.460339] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 582.265379] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 585.336707] Lustre: Failing over lustre-MDT0000 [ 585.576930] Lustre: server umount lustre-MDT0000 complete [ 586.407147] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 586.419568] LustreError: 20664:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 586.428310] Lustre: Skipped 2 previous similar messages [ 586.459129] LustreError: 20664:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 603.123400] Lustre: 3643:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788858358/real 1788858358] req@ffff968103c15880 x1875753397557376/t0(0) o400->MGC192.168.202.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788858374 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 603.155610] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 603.187237] LustreError: 19937:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 603.220351] LustreError: 19937:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 606.477138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 614.370616] LustreError: 3640:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff96823e43d500 x1875753397560576/t0(0) o250->MGC192.168.202.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 614.708109] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 614.710242] Lustre: Skipped 1 previous similar message [ 614.749779] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 619.862914] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 620.003600] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 620.009374] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 620.055550] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 630.565486] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 635.362751] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 635.382737] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 635.399159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 635.406610] Lustre: Skipped 2 previous similar messages [ 640.566140] Lustre: server umount lustre-MDT0000 complete [ 641.506884] LustreError: 19937:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 641.517887] LustreError: 19937:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 644.519350] LustreError: 19918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788858416 with bad export cookie 364388913159390386 [ 644.523888] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 644.533222] LustreError: 19918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 644.781370] Lustre: server umount lustre-MDT0001 complete [ 653.201630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 658.230365] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 668.658750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 675.445990] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 684.699917] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 691.719459] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 694.692162] Lustre: Failing over lustre-MDT0000 [ 694.753091] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 694.766457] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 694.786697] Lustre: Skipped 1 previous similar message [ 694.808693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 694.824551] Lustre: Skipped 3 previous similar messages [ 694.998682] Lustre: server umount lustre-MDT0000 complete [ 706.019515] LustreError: 23272:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 706.041955] LustreError: 23272:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 712.161581] Lustre: 3643:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788858468/real 1788858468] req@ffff96823e3f2d80 x1875753397587968/t0(0) o400->MGC192.168.202.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788858484 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 712.202862] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 715.655358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 723.427178] LustreError: 3640:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff96823e3f0700 x1875753397590912/t0(0) o250->MGC192.168.202.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 724.056126] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 729.069796] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 729.091158] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 729.101315] Lustre: Skipped 1 previous similar message [ 729.142846] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 729.516710] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 741.201086] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 746.470277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 746.479616] Lustre: Skipped 1 previous similar message [ 759.268067] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 759.718185] Lustre: server umount lustre-MDT0000 complete [ 764.076920] LustreError: 23253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788858536 with bad export cookie 364388913159391324 [ 764.097274] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 764.098389] LustreError: 23253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 764.533674] Lustre: server umount lustre-MDT0001 complete [ 774.363195] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 05:09:05 (1788858545) [ 791.510129] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 801.771207] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 846.939640] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 859.172754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 859.351631] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 859.374851] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 859.469556] Lustre: lustre-MDT0000: new disk, initializing [ 859.603251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 859.615548] Lustre: Skipped 3 previous similar messages [ 859.649275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 865.161818] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 877.505572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 877.663696] Lustre: 29659:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 878.018272] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 883.670399] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 894.515428] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 900.241559] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 907.875632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 908.274831] Lustre: 31632:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 909.346237] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 909.412569] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 909.547776] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 909.570029] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 914.732177] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 925.413695] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 931.702878] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 955.901099] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 955.906884] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 955.939586] Lustre: Skipped 3 previous similar messages [ 955.949168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 955.988488] Lustre: Skipped 5 previous similar messages [ 957.735780] Lustre: server umount lustre-OST0000 complete [ 967.978469] Lustre: server umount lustre-MDT0000 complete [ 971.235267] LustreError: 30538:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 971.257086] LustreError: 30538:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 972.369905] LustreError: 29653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788858744 with bad export cookie 364388913159391730 [ 972.387158] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 972.408329] LustreError: 29653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 973.218528] Lustre: server umount lustre-MDT0001 complete [ 989.007673] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 993.490080] Key type lgssc unregistered [ 993.864180] LNet: 33611:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 993.877187] LNetError: 33611:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 993.891110] LNet: Removed LNI 192.168.202.149@tcp [ 995.237675] Key type .llcrypt unregistered [ 995.239206] Key type ._llcrypt unregistered [ 1024.657213] Key type ._llcrypt registered [ 1024.661418] Key type .llcrypt registered [ 1024.782480] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 1026.113717] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1026.226713] alg: No test for adler32 (adler32-zlib) [ 1027.414446] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 1027.678741] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [ 1029.376245] Key type lgssc registered [ 1030.642492] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1040.415264] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1040.444419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1041.995459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1047.181515] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1052.129357] LustreError: 35022:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1052.148978] LustreError: 35022:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1056.033281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1056.469405] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1061.842690] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1070.007348] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1075.919425] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1083.047109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1083.504700] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1087.659701] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1090.155238] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1098.337981] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1100.931838] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1113.712078] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 05:14:45 (1788858885) [ 1115.646908] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1117.462937] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1124.693647] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 05:14:55 (1788858895) [ 1134.627545] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 05:15:06 (1788858906) [ 1135.771467] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1144.218883] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1144.223614] Lustre: Skipped 2 previous similar messages [ 1146.416931] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 05:15:17 (1788858917) [ 1148.616189] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1148.621922] Lustre: Skipped 2 previous similar messages [ 1158.363721] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1158.367806] Lustre: Skipped 4 previous similar messages [ 1175.257987] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 05:15:46 (1788858946) [ 1178.613761] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1183.435508] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1192.812082] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 05:16:04 (1788858964) [ 1195.380264] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1195.384337] Lustre: Skipped 6 previous similar messages [ 1210.883972] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 05:16:22 (1788858982) [ 1229.287469] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1229.290351] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1229.292928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1229.304979] Lustre: Skipped 1 previous similar message [ 1232.773469] Lustre: server umount lustre-OST0000 complete [ 1239.525559] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1239.540169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1239.542720] Lustre: Skipped 1 previous similar message [ 1243.660787] Lustre: server umount lustre-MDT0000 complete [ 1244.653139] LustreError: 35018:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1244.677267] LustreError: 35018:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1248.026495] LustreError: 35003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788859020 with bad export cookie 12480691163136985353 [ 1248.030255] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1248.035061] LustreError: 35003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1248.534302] Lustre: server umount lustre-MDT0001 complete [ 1262.641193] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 1266.926477] Key type lgssc unregistered [ 1267.278859] LNet: 42867:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1267.285604] LNetError: 42867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1267.316140] LNet: Removed LNI 192.168.202.149@tcp [ 1268.523845] Key type .llcrypt unregistered [ 1268.527064] Key type ._llcrypt unregistered [ 1289.875177] Key type ._llcrypt registered [ 1289.877415] Key type .llcrypt registered [ 1289.981569] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 1308.753337] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 1310.832093] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1310.863587] alg: No test for adler32 (adler32-zlib) [ 1312.092710] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 1312.481255] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [ 1314.264179] Key type lgssc registered [ 1315.955775] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1358.795984] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1358.856674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1360.329109] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1360.428506] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1360.597669] Lustre: lustre-MDT0000: new disk, initializing [ 1360.695883] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1360.715340] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1365.878086] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1378.927258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1379.148170] Lustre: 46020:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1379.251576] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1379.260689] Lustre: Skipped 1 previous similar message [ 1379.380653] Lustre: lustre-MDT0001: new disk, initializing [ 1379.482819] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1379.522689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1379.537697] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1385.159500] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1398.060289] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1404.733331] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1414.006792] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1414.429883] Lustre: lustre-OST0000: new disk, initializing [ 1414.439400] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1414.452980] Lustre: 47986:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1414.607415] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1422.927796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1422.939964] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1423.069782] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1423.351404] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1433.457684] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1438.085540] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1438.279912] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1443.002819] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1443.142696] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1448.936589] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1448.938200] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1448.940475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1449.004993] Lustre: Skipped 1 previous similar message [ 1451.703940] Lustre: server umount lustre-OST0000 complete [ 1456.610270] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1456.626903] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1456.644101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1456.646328] Lustre: Skipped 1 previous similar message [ 1459.172371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1459.173402] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1462.757972] Lustre: server umount lustre-MDT0000 complete [ 1466.896096] LustreError: 46012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788859239 with bad export cookie 1505271390401579216 [ 1466.905970] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1466.911319] LustreError: 46012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1467.421846] Lustre: server umount lustre-MDT0001 complete [ 1480.048397] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 05:20:51 (1788859251) [ 1489.228702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1490.458702] LustreError: 50445:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1490.482577] LustreError: 50445:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1490.565786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1496.052524] LustreError: 50446:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1496.480727] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1500.135051] LustreError: 50445:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1505.250655] LustreError: 50446:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1506.549726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1506.843370] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1512.458975] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1522.328566] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1531.820677] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1540.172779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1540.451817] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1547.556681] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1556.550712] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1566.296381] LustreError: 52979:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1567.532944] Lustre: Modifying parameter general.timeout=1 in log params [ 1568.556734] Lustre: Modifying parameter general.timeout=2 in log params [ 1571.323161] Lustre: Modifying parameter general.timeout=5 in log params [ 1571.327883] Lustre: Skipped 2 previous similar messages [ 1575.420609] Lustre: Modifying parameter general.timeout=9 in log params [ 1575.426801] Lustre: Skipped 3 previous similar messages [ 1583.425665] Lustre: Modifying parameter general.timeout=17 in log params [ 1583.432175] Lustre: Skipped 7 previous similar messages [ 1599.634222] Lustre: Modifying parameter general.timeout=34 in log params [ 1599.638488] Lustre: Skipped 16 previous similar messages [ 1632.436387] Lustre: Modifying parameter general.timeout=66 in log params [ 1632.445025] Lustre: Skipped 31 previous similar messages [ 1696.441522] Lustre: Modifying parameter general.timeout=125 in log params [ 1696.450108] Lustre: Skipped 58 previous similar messages [ 1716.496856] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 05:24:47 (1788859487) [ 1720.453539] Lustre: 59941:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1722.948180] Lustre: 60043:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1725.293711] Lustre: 60142:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1730.022492] Lustre: 60351:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1730.035309] Lustre: 60351:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1765.292853] Lustre: 61818:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1765.303381] Lustre: 61818:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1765.361840] LustreError: 61818:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1770.247397] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 05:25:41 (1788859541) [ 1786.338208] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1786.357902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1786.366786] Lustre: Skipped 2 previous similar messages [ 1789.866599] Lustre: server umount lustre-MDT0000 complete [ 1791.466459] LustreError: 51173:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1791.497318] LustreError: 51173:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1794.908856] LustreError: 55308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788859567 with bad export cookie 1505271390401580098 [ 1794.914497] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1794.919512] LustreError: 55308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1795.278655] Lustre: server umount lustre-MDT0001 complete [ 1810.897372] Lustre: server umount lustre-OST0000 complete [ 1844.507252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1844.798433] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1844.845986] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1844.851858] Lustre: Skipped 27 previous similar messages [ 1845.056566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1849.731375] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1859.005418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1859.140044] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1859.190647] Lustre: 64353:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1864.157793] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1872.355640] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1878.740042] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1888.148107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1888.335682] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1888.722452] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1888.731511] Lustre: Skipped 1 previous similar message [ 1895.566843] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1904.616716] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1906.776963] Lustre: 66923:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1914.853125] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1914.859305] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1914.879405] Lustre: Skipped 2 previous similar messages [ 1914.891135] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1920.208900] Lustre: server umount lustre-OST0000 complete [ 1925.606844] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1925.616545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1925.626057] Lustre: Skipped 2 previous similar messages [ 1925.652211] Lustre: Skipped 3 previous similar messages [ 1931.710683] Lustre: server umount lustre-MDT0000 complete [ 1935.842173] LustreError: 64365:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1935.853720] LustreError: 64365:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1936.382281] LustreError: 65096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788859708 with bad export cookie 1505271390401583367 [ 1936.394377] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1936.793841] Lustre: server umount lustre-MDT0001 complete [ 1950.884679] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 1954.072614] Key type lgssc unregistered [ 1954.507601] LNet: 68131:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1954.515824] LNetError: 68131:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1954.525616] LNet: Removed LNI 192.168.202.149@tcp [ 1955.634609] Key type .llcrypt unregistered [ 1955.640119] Key type ._llcrypt unregistered [ 1986.096745] Key type ._llcrypt registered [ 1986.098537] Key type .llcrypt registered [ 1986.181546] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 1987.635593] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1987.661256] alg: No test for adler32 (adler32-zlib) [ 1988.874289] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 1989.174778] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [ 1990.864282] Key type lgssc registered [ 1992.127163] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2000.948368] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2000.998352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2002.756334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2006.469290] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2012.138175] LustreError: 69541:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2012.164218] LustreError: 69541:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2017.268111] LustreError: 69540:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2017.781040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2018.747480] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2024.174671] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2032.569693] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2038.956896] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2045.731518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2046.145455] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2053.622366] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2063.049258] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2071.017675] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2084.329786] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2084.336431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2084.362866] Lustre: Skipped 1 previous similar message [ 2084.378582] Lustre: Skipped 2 previous similar messages [ 2089.442706] LustreError: 69540:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2089.475269] LustreError: 69540:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2089.490548] Lustre: server umount lustre-MDT0000 complete [ 2094.249244] LustreError: 69522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788859866 with bad export cookie 16692988433156429222 [ 2094.270124] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2094.288636] LustreError: 69522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2094.565351] LustreError: 71679:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2094.580887] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2094.590889] Lustre: Skipped 1 previous similar message [ 2094.911146] Lustre: server umount lustre-MDT0001 complete [ 2102.259065] Lustre: server umount lustre-OST0000 complete [ 2130.570995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2130.987316] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2131.068398] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2131.611624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2135.987272] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2145.266414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2145.315411] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2145.332897] Lustre: 74413:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2145.653963] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2150.528958] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2159.875917] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2167.404886] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2175.947218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2176.131834] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2176.484261] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2183.508582] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2192.264069] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2194.161768] Lustre: 76988:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2196.756615] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2204.641571] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2204.662652] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2204.729614] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2205.157143] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2210.100759] Lustre: server umount lustre-OST0000 complete [ 2215.393543] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2215.396880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2215.417555] Lustre: Skipped 2 previous similar messages [ 2215.432322] Lustre: Skipped 3 previous similar messages [ 2220.513824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2220.520524] Lustre: Skipped 1 previous similar message [ 2221.748864] Lustre: server umount lustre-MDT0000 complete [ 2225.639545] LustreError: 74423:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2225.678232] LustreError: 74423:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2225.747745] LustreError: 74404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788859997 with bad export cookie 16692988433156430069 [ 2225.748417] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2225.770683] LustreError: 74404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2226.215987] Lustre: server umount lustre-MDT0001 complete [ 2238.936711] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 2242.269795] Key type lgssc unregistered [ 2242.600362] LNet: 78201:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2242.612324] LNetError: 78201:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2242.623933] LNet: Removed LNI 192.168.202.149@tcp [ 2243.653865] Key type .llcrypt unregistered [ 2243.656884] Key type ._llcrypt unregistered [ 2270.497865] Key type ._llcrypt registered [ 2270.501315] Key type .llcrypt registered [ 2270.606865] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 2271.705828] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2271.727662] alg: No test for adler32 (adler32-zlib) [ 2272.977295] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 2273.261792] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [ 2274.920234] Key type lgssc registered [ 2276.450676] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2284.790562] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2284.811237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2286.381373] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2292.101269] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2296.813028] LustreError: 79611:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2296.843475] LustreError: 79611:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2301.922584] LustreError: 79610:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2302.368474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2302.823285] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2308.110759] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2317.271073] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2323.420334] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2329.767353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2330.206701] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2337.302435] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2344.614586] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2355.169275] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2356.374275] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2357.504329] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2359.550801] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2359.554724] Lustre: Skipped 1 previous similar message [ 2363.744846] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2363.753878] Lustre: Skipped 3 previous similar messages [ 2372.823855] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2372.830294] Lustre: Skipped 8 previous similar messages [ 2388.995593] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2389.000995] Lustre: Skipped 15 previous similar messages [ 2421.815769] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2421.818400] Lustre: Skipped 31 previous similar messages [ 2466.247309] Lustre: 87714:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2469.054230] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 05:37:20 (1788860240) [ 2470.673262] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2472.752859] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 05:37:24 (1788860244) [ 2479.025325] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2479.037330] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2479.057974] Lustre: Skipped 1 previous similar message [ 2479.086225] Lustre: Skipped 1 previous similar message [ 2483.857378] Lustre: server umount lustre-OST0000 complete [ 2489.325375] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2489.347936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2489.376034] Lustre: Skipped 1 previous similar message [ 2494.433681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2494.440847] Lustre: Skipped 1 previous similar message [ 2497.509810] Lustre: server umount lustre-MDT0000 complete [ 2499.561364] LustreError: 81049:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2499.576576] LustreError: 81049:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2502.533813] LustreError: 82198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788860274 with bad export cookie 4484193117659343653 [ 2502.534437] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2502.539922] LustreError: 82198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2502.878623] Lustre: server umount lustre-MDT0001 complete [ 2513.869698] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 2516.747759] Key type lgssc unregistered [ 2517.250183] LNet: 88962:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2517.261669] LNetError: 88962:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2517.285982] LNet: Removed LNI 192.168.202.149@tcp [ 2518.676316] Key type .llcrypt unregistered [ 2518.678166] Key type ._llcrypt unregistered [ 2523.601862] Key type ._llcrypt registered [ 2523.605395] Key type .llcrypt registered [ 2523.751782] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2528.522457] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules [ 2529.665215] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2529.750470] alg: No test for adler32 (adler32-zlib) [ 2530.735718] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 2531.104375] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [ 2532.849157] Key type lgssc registered [ 2534.599647] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2536.219381] LustreError: 90179:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2554.580413] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 2557.486765] LustreError: 90179:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2564.363218] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2564.386031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2565.944026] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2571.414626] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2576.367361] LustreError: 91160:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2576.385524] LustreError: 91160:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2581.476584] LustreError: 91159:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2581.838377] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 05:39:13 (1788860353) [ 2586.602890] LustreError: 91160:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2587.591055] LustreError: 91191:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2587.600414] LustreError: 91191:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2587.608504] LustreError: 91191:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 2588.088923] Lustre: server umount lustre-MDT0000 complete [ 2603.043697] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 2606.677167] Key type lgssc unregistered [ 2607.175318] LNet: 92685:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2607.206531] LNetError: 92685:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2607.228322] LNet: Removed LNI 192.168.202.149@tcp [ 2608.443076] Key type .llcrypt unregistered [ 2608.444759] Key type ._llcrypt unregistered [ 2630.902352] Key type ._llcrypt registered [ 2630.904772] Key type .llcrypt registered [ 2631.073926] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 2632.507887] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2632.533677] alg: No test for adler32 (adler32-zlib) [ 2633.710511] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 2634.046819] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [ 2635.859895] Key type lgssc registered [ 2637.185704] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2645.052541] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2645.103353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2646.610222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2651.391600] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2656.737461] LustreError: 93898:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2656.755326] LustreError: 93898:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2660.831143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2661.927381] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2666.794301] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2676.124680] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2682.592832] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2689.943892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2691.069386] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2697.703319] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2706.025963] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2755.934409] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 05:42:07 (1788860527) [ 2758.202477] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2760.331720] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 05:42:11 (1788860531) [ 2767.855026] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2767.862498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2767.867318] Lustre: Skipped 1 previous similar message [ 2767.878397] Lustre: Skipped 2 previous similar messages [ 2771.176686] Lustre: server umount lustre-MDT0000 complete [ 2772.978372] LustreError: 93899:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2772.999811] LustreError: 93899:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2775.780602] LustreError: 93879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788860547 with bad export cookie 8801656747028688886 [ 2775.797933] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2775.813032] LustreError: 93879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2776.252896] Lustre: server umount lustre-MDT0001 complete [ 2790.926803] Lustre: server umount lustre-OST0000 complete [ 2802.909282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2803.523371] LustreError: 98174:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2803.896226] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2808.755641] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2809.330184] LustreError: 98175:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2813.421748] LustreError: 98174:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2818.181986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2818.617328] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2824.054585] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2831.415828] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2838.731453] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2853.061117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2853.453674] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2853.469469] LustreError: 98164:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2853.490321] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2853.495567] LustreError: 100258:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2853.500811] LustreError: 100258:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2853.504678] LustreError: 100258:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2853.570526] Lustre: server umount lustre-OST0000 complete [ 2853.573533] LustreError: 100258:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2862.085699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2862.502598] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2862.513827] LustreError: 98165:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2862.602136] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2862.612785] LustreError: 100725:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2862.643700] LustreError: 100725:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2862.654159] LustreError: 100725:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2862.780888] Lustre: server umount lustre-OST0000 complete [ 2862.788534] LustreError: 100725:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2872.120246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2872.260573] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2872.264431] Lustre: Found index 0 for lustre-OST0000, updating log [ 2872.268099] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2872.301548] Lustre: lustre-OST0000: new disk, initializing [ 2872.304227] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2872.309569] Lustre: 101260:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2872.514619] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2878.273115] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2881.700638] Lustre: Failing over lustre-OST0000 [ 2881.863962] Lustre: server umount lustre-OST0000 complete [ 2887.843549] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2887.846213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2887.864666] Lustre: Skipped 2 previous similar messages [ 2893.291699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2893.301237] Lustre: Skipped 2 previous similar messages [ 2895.431243] Lustre: server umount lustre-MDT0000 complete [ 2898.402207] LustreError: 98174:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2898.429250] LustreError: 98174:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2899.292285] LustreError: 101259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788860671 with bad export cookie 8801656747028691623 [ 2899.297636] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2899.306078] LustreError: 101259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 2899.556879] Lustre: server umount lustre-MDT0001 complete [ 2909.160740] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 05:44:40 (1788860680) [ 2926.870290] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 2936.592142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2936.996167] LustreError: 103607:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2937.165046] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2940.976220] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2948.689836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2955.261275] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2959.095200] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2965.665227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2966.308536] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2974.504648] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2974.511706] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2974.545601] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2975.164183] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2979.830751] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 2979.875646] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2981.932875] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:02 [ 2981.970844] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2982.000421] Lustre: lustre-OST0000: Recovery over after 0:07, of 1 clients 1 recovered and 0 were evicted. [ 2982.003276] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2982.016386] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2983.994456] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2984.125185] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2984.274677] Lustre: lustre-OST0001: new disk, initializing [ 2984.282959] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2984.293938] Lustre: 105928:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2984.817972] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2984.820755] Lustre: Skipped 2 previous similar messages [ 2992.332138] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2994.321065] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2994.338135] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2994.457355] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3001.501963] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3007.933933] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3015.146552] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3015.158087] Lustre: Skipped 1 previous similar message [ 3015.164504] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3015.171022] Lustre: Skipped 2 previous similar messages [ 3018.646846] Lustre: server umount lustre-MDT0001 complete [ 3020.285176] LustreError: 104339:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3020.315165] LustreError: 104339:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 3030.529335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3030.742404] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3030.748263] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3030.881714] Lustre: 103595:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3032.532850] LustreError: 107469:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3032.543785] LustreError: 107469:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3032.553337] LustreError: 103600:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3032.752811] LustreError: 107478:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.149@tcp: cfg command failed: rc = -17 [ 3032.763527] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3032.763527] [ 3037.200711] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3037.868181] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3049.103320] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 05:47:00 (1788860820) [ 3069.273519] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 3089.590693] Lustre: 109547:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3113.517318] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3117.365251] Lustre: 110682:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3195.371333] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3195.380279] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3195.397770] Lustre: Skipped 1 previous similar message [ 3195.401037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3195.408617] Lustre: Skipped 1 previous similar message [ 3201.305038] Lustre: server umount lustre-MDT0000 complete [ 3205.842936] LustreError: 106647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788860978 with bad export cookie 8801656747028692631 [ 3205.863783] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3205.865279] LustreError: 106647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3206.118469] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3206.120502] LustreError: 105939:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3206.126681] Lustre: Skipped 3 previous similar messages [ 3206.127943] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3206.127949] Lustre: Skipped 7 previous similar messages [ 3206.168811] LustreError: 105939:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 3206.714591] Lustre: server umount lustre-MDT0001 complete [ 3213.828376] Lustre: server umount lustre-OST0000 complete [ 3218.545481] Lustre: server umount lustre-OST0001 complete [ 3227.325821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3240.654154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3250.581762] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3262.899798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3295.095747] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 3311.570380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3311.594109] Lustre: lustre-MDT0000: reset Object Index mappings [ 3312.450848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3312.469434] Lustre: Skipped 1 previous similar message [ 3318.678386] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3332.333642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3332.376545] Lustre: lustre-MDT0001: reset Object Index mappings [ 3341.028948] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3344.951158] Lustre: 115806:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3353.710063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3357.771962] LustreError: 116163:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3357.796046] LustreError: 116163:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 3357.799206] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:381 to 0x280000401:418) [ 3358.890414] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 3362.951765] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3373.121563] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3379.204071] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3379.206970] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3382.105124] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3391.263375] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3395.356218] Lustre: 117683:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3447.816773] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 05:53:38 (1788861218) [ 3476.449881] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3476.457138] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3476.461067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3476.469914] Lustre: Skipped 4 previous similar messages [ 3476.472633] Lustre: Skipped 3 previous similar messages [ 3478.580650] Lustre: server umount lustre-MDT0000 complete [ 3482.918335] LustreError: 114636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788861255 with bad export cookie 8801656747028739727 [ 3482.920307] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3482.945624] LustreError: 114636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3489.767256] Lustre: server umount lustre-MDT0001 complete [ 3501.750170] Lustre: server umount lustre-OST0000 complete [ 3512.962629] Lustre: server umount lustre-OST0001 complete [ 3521.689925] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 3536.779221] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 3582.463609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3582.712923] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3582.822634] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3582.931363] Lustre: lustre-MDT0000: new disk, initializing [ 3583.012520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3583.032261] Lustre: Skipped 3 previous similar messages [ 3583.050872] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3589.990196] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3596.826722] Lustre: server umount lustre-MDT0000 complete [ 3606.394355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3606.652741] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3606.670838] Lustre: 121967:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3606.684878] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3611.744823] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3617.427440] Lustre: server umount lustre-MDT0000 complete [ 3627.586366] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 05:56:39 (1788861399) [ 3629.545129] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3631.455298] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 05:56:43 (1788861403) [ 3651.697686] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 3665.635615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3670.649731] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3681.445611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3681.685838] Lustre: 124019:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3681.807964] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3681.818284] Lustre: Skipped 1 previous similar message [ 3681.940360] Lustre: lustre-MDT0001: new disk, initializing [ 3682.141547] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3682.164865] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3687.654573] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3694.032574] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3701.656987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3702.006500] Lustre: lustre-OST0000: new disk, initializing [ 3702.009997] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3702.016659] Lustre: 125598:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3703.618903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3703.641742] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3703.801500] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3708.980188] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3720.769524] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3721.035573] Lustre: lustre-OST0001: new disk, initializing [ 3721.038414] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3721.043394] Lustre: 126470:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3722.237406] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3722.246285] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3722.437400] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3729.463636] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3739.142827] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3743.293445] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4150.125284] Lustre: 132893:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4150.138218] Lustre: 132893:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4150.144364] Lustre: 132893:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4150.150646] Lustre: 132893:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4150.156234] Lustre: 132893:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4150.161531] Lustre: 132893:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4450.659437] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 06:10:21 (1788862221) [ 4468.197605] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4468.203582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4468.231631] Lustre: Skipped 5 previous similar messages [ 4468.244505] Lustre: Skipped 7 previous similar messages [ 4472.280120] Lustre: server umount lustre-MDT0000 complete [ 4473.318232] LustreError: 125591:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4473.344183] LustreError: 125591:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 17 previous similar messages [ 4477.354632] LustreError: 124010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788862249 with bad export cookie 8801656747028807564 [ 4477.364466] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4477.376367] LustreError: 124010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4478.435124] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4478.467301] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4478.477230] Lustre: Skipped 1 previous similar message [ 4483.938721] Lustre: server umount lustre-MDT0001 complete [ 4495.932202] Lustre: server umount lustre-OST0000 complete [ 4515.296488] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4515.658286] Lustre: server umount lustre-OST0001 complete [ 4523.188913] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 4533.250197] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 4569.269876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4569.541231] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4569.574335] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4569.681086] Lustre: lustre-MDT0000: new disk, initializing [ 4569.767410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4569.773317] Lustre: Skipped 5 previous similar messages [ 4569.785614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4575.427874] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 4586.924605] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4587.115508] Lustre: 135948:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4587.188614] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4587.194689] Lustre: Skipped 1 previous similar message [ 4587.275196] Lustre: lustre-MDT0001: new disk, initializing [ 4587.498468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4587.505220] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4593.256621] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 4603.656805] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4610.213471] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4618.322464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4618.840243] Lustre: lustre-OST0000: new disk, initializing [ 4618.861817] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4618.869938] Lustre: 137915:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4619.027953] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4619.044660] Lustre: Skipped 1 previous similar message [ 4620.307606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4620.325724] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4620.509030] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4627.310578] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 4639.005105] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4650.694812] Lustre: lustre-MDD0000: changelog on [ 4653.705301] Lustre: lustre-MDD0001: changelog on [ 6519.654410] Lustre: lustre-MDD0001: changelog off [ 6521.790526] Lustre: 135957:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6522.768412] Lustre: lustre-MDD0000: changelog off [ 6530.532704] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6530.534474] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6530.539891] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6530.539900] Lustre: Skipped 2 previous similar messages [ 6530.547388] Lustre: Skipped 2 previous similar messages [ 6535.660448] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6535.670508] Lustre: Skipped 2 previous similar messages [ 6540.771013] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6540.775029] Lustre: Skipped 1 previous similar message [ 6544.352315] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6544.759198] Lustre: server umount lustre-OST0000 complete [ 6551.032299] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6551.038525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6551.045277] Lustre: Skipped 1 previous similar message [ 6551.064795] Lustre: Skipped 1 previous similar message [ 6561.249698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6561.255273] Lustre: Skipped 3 previous similar messages [ 6563.296293] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6563.781638] Lustre: server umount lustre-MDT0000 complete [ 6566.381549] LustreError: 135960:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6566.410692] LustreError: 135960:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6567.778232] LustreError: 135940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788864339 with bad export cookie 8801656747029623281 [ 6567.782253] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6567.785096] LustreError: 135940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6568.248399] Lustre: server umount lustre-MDT0001 complete [ 6580.029972] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 6583.159268] Key type lgssc unregistered [ 6583.427815] LNet: 147046:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6583.442213] LNetError: 147046:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6583.465828] LNet: Removed LNI 192.168.202.149@tcp [ 6584.334238] Key type .llcrypt unregistered [ 6584.339401] Key type ._llcrypt unregistered [ 6611.238710] Key type ._llcrypt registered [ 6611.244487] Key type .llcrypt registered [ 6611.450577] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 6628.841668] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 6630.803366] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6630.872082] alg: No test for adler32 (adler32-zlib) [ 6632.302946] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 6632.782101] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [ 6634.504346] Key type lgssc registered [ 6635.962589] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6680.096648] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6680.137611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6681.663421] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6681.716348] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6681.906941] Lustre: lustre-MDT0000: new disk, initializing [ 6682.023683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6682.064515] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6688.531352] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 6701.693716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6701.863215] Lustre: 150245:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6701.928907] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6701.933285] Lustre: Skipped 1 previous similar message [ 6702.105714] Lustre: lustre-MDT0001: new disk, initializing [ 6702.258312] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6702.318487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6702.329872] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6707.521419] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 6719.456388] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6725.692723] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6733.991878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6734.468794] Lustre: lustre-OST0000: new disk, initializing [ 6734.476913] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6734.498316] Lustre: 152218:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6734.704048] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6740.629500] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6740.657520] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6740.873350] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6743.852367] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 6755.577721] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6762.042304] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_module obdecho/obdecho [ 6764.575845] LustreError: 153223:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6764.587582] LustreError: 153223:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6767.079700] LustreError: 153319:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6781.926493] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6781.939985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6781.948379] Lustre: Skipped 1 previous similar message [ 6781.970161] Lustre: Skipped 1 previous similar message [ 6786.922526] Lustre: server umount lustre-MDT0000 complete [ 6791.721427] LustreError: 150237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788864563 with bad export cookie 14963840220972139127 [ 6791.738909] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6791.746976] LustreError: 150237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6792.141762] Lustre: server umount lustre-MDT0001 complete [ 6806.402712] Lustre: server umount lustre-OST0000 complete [ 6832.883469] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 6843.295748] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 6896.956015] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6897.369426] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6897.441448] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6897.603988] Lustre: lustre-MDT0000: new disk, initializing [ 6897.798745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6897.819275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6904.174308] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 6917.599415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6917.868672] Lustre: 157036:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6917.964283] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6917.967934] Lustre: Skipped 1 previous similar message [ 6918.023547] Lustre: lustre-MDT0001: new disk, initializing [ 6918.126589] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6918.170931] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6918.202916] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6923.821844] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 6933.648169] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6940.463525] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6948.212765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6948.503519] Lustre: lustre-OST0000: new disk, initializing [ 6948.507156] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6948.519164] Lustre: 159006:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6948.615299] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6955.595528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6955.605101] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6955.751796] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6955.950773] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 6967.991870] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6974.107294] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6974.484540] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6980.071982] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6980.449994] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6986.725539] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6986.726452] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6986.728654] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6986.753541] Lustre: Skipped 2 previous similar messages [ 6986.780110] LustreError: Skipped 1 previous similar message [ 6989.131249] Lustre: server umount lustre-OST0000 complete [ 6994.914637] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6994.924169] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6994.938979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6994.945174] Lustre: Skipped 1 previous similar message [ 6996.964261] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6996.965988] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6996.985470] Lustre: Skipped 1 previous similar message [ 6999.652688] Lustre: server umount lustre-MDT0000 complete [ 7002.105506] LustreError: 157049:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7002.130109] LustreError: 157049:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 7004.242139] LustreError: 157029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788864776 with bad export cookie 14963840220972140093 [ 7004.248724] LustreError: 157029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7004.254221] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7004.705850] Lustre: server umount lustre-MDT0001 complete [ 7008.971342] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 06:53:00 (1788864780) [ 7010.901804] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7013.015995] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 06:53:04 (1788864784) [ 7021.429587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7022.035460] LustreError: 161418:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7022.181864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7027.196668] LustreError: 161418:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7028.171121] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7032.315199] LustreError: 161419:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7038.406230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7038.434155] LustreError: 161418:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7038.458981] LustreError: 161418:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7043.242543] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7052.288906] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7058.372310] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7066.861465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7067.585153] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7067.596573] Lustre: Skipped 1 previous similar message [ 7075.412701] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7082.538523] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7093.231424] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7093.256084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7093.273524] Lustre: Skipped 2 previous similar messages [ 7093.320504] Lustre: Skipped 2 previous similar messages [ 7097.244741] Lustre: server umount lustre-MDT0000 complete [ 7098.342121] LustreError: 161419:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7098.358023] LustreError: 161419:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7101.757956] LustreError: 161399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788864873 with bad export cookie 14963840220972140975 [ 7101.775378] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7101.782948] LustreError: 161399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 7103.460523] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7103.476295] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7103.482671] Lustre: Skipped 1 previous similar message [ 7116.256133] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7116.615075] Lustre: server umount lustre-MDT0001 complete [ 7123.946739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7137.989478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7138.309727] LustreError: 165264:0:(llog.c:1655:llog_backup()) MGC192.168.202.149@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7138.317674] Lustre: 165264:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.149@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7142.371579] LustreError: 165269:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7142.386996] LustreError: 165269:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 7177.187736] LustreError: 165289:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7177.207160] LustreError: 165289:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 28 previous similar messages [ 7179.314844] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.49@tcp (not set up) [ 7179.325240] Lustre: Skipped 6 previous similar messages [ 7199.520265] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7199.525543] Lustre: 165264:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.149@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7240.210372] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7244.855261] LustreError: 165270:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7244.876199] LustreError: 165270:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 42 previous similar messages [ 7245.320149] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7246.520402] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7258.178181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7267.815792] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7271.945035] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xcfaa3d662ebf6308 [ 7271.962215] Lustre: MGC192.168.202.149@tcp: Connection restored to 0@lo (at 0@lo) [ 7277.578560] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7279.283182] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7289.462629] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7291.639951] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7298.651629] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7301.092624] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7323.618856] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7323.619786] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7323.630554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7323.676481] Lustre: Skipped 2 previous similar messages [ 7330.228933] Lustre: server umount lustre-MDT0000 complete [ 7334.559832] LustreError: 164204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788865106 with bad export cookie 14963840220972147464 [ 7334.571415] LustreError: 164204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7334.575290] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7335.144543] Lustre: server umount lustre-MDT0001 complete [ 7350.370432] Lustre: server umount lustre-OST0000 complete [ 7378.502602] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 7394.192865] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 7440.219845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7440.387629] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7440.407288] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7440.510730] Lustre: lustre-MDT0000: new disk, initializing [ 7440.647640] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7440.659973] Lustre: Skipped 1 previous similar message [ 7440.686740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7446.920574] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7459.889749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7460.120722] Lustre: 170399:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7460.438240] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7466.145237] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7480.265251] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7487.469208] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7497.342889] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7497.827554] Lustre: lustre-OST0000: new disk, initializing [ 7497.832324] Lustre: Skipped 1 previous similar message [ 7497.838063] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7497.845974] Lustre: Skipped 2 previous similar messages [ 7497.853709] Lustre: 172379:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7502.939182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7502.944836] Lustre: Skipped 1 previous similar message [ 7502.955123] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7503.083402] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7506.474855] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7519.241545] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7525.676777] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7526.061264] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7532.732649] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7533.074954] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7539.182488] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7539.190992] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7539.192890] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7539.192898] Lustre: Skipped 5 previous similar messages [ 7539.221236] Lustre: Skipped 1 previous similar message [ 7539.234851] LustreError: Skipped 1 previous similar message [ 7542.310573] Lustre: server umount lustre-OST0000 complete [ 7547.366280] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7553.646521] Lustre: server umount lustre-MDT0000 complete [ 7554.530656] LustreError: 170405:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7554.561475] LustreError: 170405:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 30 previous similar messages [ 7558.773096] LustreError: 172378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788865330 with bad export cookie 14963840220972164271 [ 7558.789541] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7558.791768] LustreError: 172378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7559.210953] Lustre: server umount lustre-MDT0001 complete [ 7570.337673] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 07:02:21 (1788865341) [ 7578.727629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7586.314958] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7597.270145] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7604.212542] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7614.399924] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7623.755960] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7631.912474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7640.669546] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7650.740236] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7663.909929] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 07:03:55 (1788865435) [ 7670.241603] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7670.247993] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7670.270640] Lustre: Skipped 2 previous similar messages [ 7670.276207] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7670.278818] Lustre: Skipped 4 previous similar messages [ 7676.332956] Lustre: server umount lustre-OST0000 complete [ 7688.001155] Lustre: server umount lustre-MDT0000 complete [ 7693.336568] LustreError: 174775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788865465 with bad export cookie 14963840220972165153 [ 7693.351939] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7693.366215] LustreError: 174775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7693.885200] Lustre: server umount lustre-MDT0001 complete [ 7709.522975] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 7714.147751] Key type lgssc unregistered [ 7714.580346] LNet: 178526:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7714.595287] LNetError: 178526:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7714.614155] LNet: Removed LNI 192.168.202.149@tcp [ 7715.917369] Key type .llcrypt unregistered [ 7715.919445] Key type ._llcrypt unregistered [ 7740.082753] Key type ._llcrypt registered [ 7740.084214] Key type .llcrypt registered [ 7740.287604] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 7741.741991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7741.865254] alg: No test for adler32 (adler32-zlib) [ 7742.950264] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 7743.180428] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [ 7745.019779] Key type lgssc registered [ 7746.769787] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7756.071670] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7756.120964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7771.809604] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7776.928349] LustreError: 179768:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.149@tcp: failed processing log, type 4: rc = -110 [ 7782.048703] LustreError: 179768:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7782.062027] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7782.069977] LustreError: MGC192.168.202.149@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7782.083338] LustreError: 179768:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7782.090769] LustreError: 179768:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -22 [ 7782.097086] LustreError: 179768:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 7782.191674] Lustre: server umount lustre-OST0000 complete [ 7782.196444] LustreError: 179768:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7792.316753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7793.776476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7800.536648] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7802.867423] LustreError: 180267:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7802.887487] LustreError: 180267:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7807.976213] LustreError: 180268:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7810.388743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7811.044138] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7817.417321] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7824.528078] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7831.286436] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7837.147854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7837.256782] LustreError: 182086:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7837.261311] Lustre: 182086:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.202.149@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7837.342390] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7844.416805] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7852.565827] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7856.162327] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7856.167232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7856.179933] Lustre: Skipped 2 previous similar messages [ 7856.202630] Lustre: Skipped 2 previous similar messages [ 7861.222231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7861.238038] Lustre: Skipped 2 previous similar messages [ 7861.771051] Lustre: server umount lustre-MDT0000 complete [ 7865.619993] LustreError: 180248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788865637 with bad export cookie 2231218329100591929 [ 7865.627956] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7865.638628] LustreError: 180248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7866.342221] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7866.342339] LustreError: 180262:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7866.368920] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7866.464428] LustreError: 180998:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000001e710c44 x1875761446287744/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7866.722465] Lustre: server umount lustre-MDT0001 complete [ 7872.920418] Lustre: server umount lustre-OST0000 complete [ 7883.476615] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 7896.164492] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 7939.267034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7939.576759] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7939.594710] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7939.654912] Lustre: lustre-MDT0000: new disk, initializing [ 7939.709955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7939.722170] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7944.830578] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7956.932939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7957.182464] Lustre: 185608:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7957.251732] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7957.255581] Lustre: Skipped 1 previous similar message [ 7957.316728] Lustre: lustre-MDT0001: new disk, initializing [ 7957.401452] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7957.441462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7957.455994] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7963.001278] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7973.986593] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7980.600651] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7987.916844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7988.221531] Lustre: lustre-OST0000: new disk, initializing [ 7988.224744] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7988.233846] Lustre: 187575:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7988.335686] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7995.474759] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7995.484518] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7995.701856] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7995.758401] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8006.882641] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8012.529623] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8012.946320] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8018.857421] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8019.187774] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8026.595738] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8026.596304] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8026.599774] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8026.627536] Lustre: Skipped 1 previous similar message [ 8026.664532] LustreError: Skipped 1 previous similar message [ 8031.717970] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8031.734935] Lustre: Skipped 1 previous similar message [ 8036.324500] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8036.477621] Lustre: server umount lustre-OST0000 complete [ 8041.953970] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8041.955271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8041.969133] Lustre: Skipped 1 previous similar message [ 8041.977685] Lustre: Skipped 1 previous similar message [ 8046.944607] Lustre: server umount lustre-MDT0000 complete [ 8047.076288] LustreError: 185615:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8051.248718] LustreError: 185600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788865823 with bad export cookie 2231218329100592650 [ 8051.273869] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8051.684021] Lustre: server umount lustre-MDT0001 complete [ 8062.396716] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 07:10:33 (1788865833) [ 8084.627361] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 8087.881612] Key type lgssc unregistered [ 8088.217851] LNet: 190524:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8088.235730] LNetError: 190524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8088.253907] LNet: Removed LNI 192.168.202.149@tcp [ 8089.432994] Key type .llcrypt unregistered [ 8089.434937] Key type ._llcrypt unregistered [ 8110.844959] Key type ._llcrypt registered [ 8110.851253] Key type .llcrypt registered [ 8110.967754] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 8112.398443] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8112.589095] alg: No test for adler32 (adler32-zlib) [ 8113.716189] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 8114.012981] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [ 8115.705257] Key type lgssc registered [ 8117.429674] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8126.253474] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8126.277955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8128.001961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8133.437742] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8138.215692] LustreError: 191739:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8138.270292] LustreError: 191739:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8143.346987] LustreError: 191740:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8146.134890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8146.535183] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8152.421863] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8161.414850] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8167.037753] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8174.962059] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8175.152813] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8175.157902] LustreError: 193605:0:(llog.c:1696:llog_backup()) MGC192.168.202.149@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8175.167435] Lustre: 193605:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.202.149@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8175.292682] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8181.356722] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8189.113336] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8192.099873] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8192.113286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8192.116713] Lustre: Skipped 1 previous similar message [ 8192.132285] Lustre: Skipped 2 previous similar messages [ 8197.604340] LustreError: 191734:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8197.639234] LustreError: 191734:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8197.711474] Lustre: server umount lustre-MDT0000 complete [ 8201.585473] LustreError: 191720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788865973 with bad export cookie 9264879966060417786 [ 8201.602085] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8202.400343] LustreError: 192469:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000006287067a x1875761835306880/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8202.590138] Lustre: server umount lustre-MDT0001 complete [ 8217.310800] Lustre: server umount lustre-OST0000 complete [ 8227.343158] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 8237.288466] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 8274.887584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8275.277780] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8275.306233] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8275.443742] Lustre: lustre-MDT0000: new disk, initializing [ 8275.601106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8275.634626] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8280.638879] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8292.099526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8292.177167] Lustre: 197132:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8292.245575] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8292.254418] Lustre: Skipped 1 previous similar message [ 8292.259281] LustreError: 197142:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8292.262936] LustreError: 197914:0:(llog.c:1696:llog_backup()) MGC192.168.202.149@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8292.262945] LustreError: 197914:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8292.262982] Lustre: 197914:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.149@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8292.282821] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8292.307913] Lustre: Skipped 1 previous similar message [ 8292.380632] Lustre: lustre-MDT0001: new disk, initializing [ 8292.500637] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8292.557794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8292.577343] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8297.428599] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8308.442759] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8314.719496] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8321.569053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8321.795126] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8321.799415] Lustre: Skipped 1 previous similar message [ 8321.802477] LustreError: 199077:0:(llog.c:1696:llog_backup()) MGC192.168.202.149@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8321.807356] LustreError: 199077:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8321.809995] Lustre: 199077:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.149@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8321.819117] Lustre: 199077:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8321.836636] Lustre: lustre-OST0000: new disk, initializing [ 8321.839354] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8321.848414] Lustre: 199102:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8321.926611] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8327.826815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8327.844682] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8328.010284] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8329.104157] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8338.967487] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8343.896950] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8344.080563] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8348.981565] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8349.206288] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8353.761908] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8353.774031] Lustre: Skipped 1 previous similar message [ 8353.780785] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8353.783997] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8353.803097] Lustre: Skipped 1 previous similar message [ 8357.732865] Lustre: server umount lustre-OST0000 complete [ 8364.005368] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8364.021383] Lustre: Skipped 1 previous similar message [ 8364.027996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8364.040148] Lustre: Skipped 1 previous similar message [ 8368.407611] Lustre: server umount lustre-MDT0000 complete [ 8369.122827] LustreError: 197139:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8369.153934] LustreError: 197139:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8372.150261] LustreError: 197122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788866144 with bad export cookie 9264879966060418500 [ 8372.169173] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8372.174562] LustreError: 197122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 8372.515123] Lustre: server umount lustre-MDT0001 complete [ 8381.070097] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 07:15:52 (1788866152) [ 8399.519576] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 8410.274513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8410.731925] LustreError: 202229:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8410.757830] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8415.876831] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8424.181083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8429.227256] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8432.545241] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8439.047270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8445.686726] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8454.954238] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8455.264569] Lustre: lustre-OST0001: new disk, initializing [ 8455.277508] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8455.292914] Lustre: 204538:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8455.435115] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8455.441775] Lustre: Skipped 2 previous similar messages [ 8461.521361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8461.534082] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8461.646991] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8462.654334] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8473.998466] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8480.474432] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8482.576530] Lustre: DEBUG MARKER: ADD OST3 [ 8488.151311] loop: module loaded [ 8488.285931] loop0: detected capacity change from 0 to 204800000 [ 8488.569941] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8488.582759] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8488.602834] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8494.629603] loop0: detected capacity change from 0 to 204800000 [ 8495.761155] Lustre: DEBUG MARKER: STOP OST3 [ 8502.011558] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8502.358564] Lustre: lustre-OST0003: new disk, initializing [ 8502.373621] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8502.402397] Lustre: 206538:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8503.775153] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8503.913575] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8503.927924] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8503.979466] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8505.824125] LustreError: 206640:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8506.515100] Lustre: Failing over lustre-OST0003 [ 8506.671930] Lustre: server umount lustre-OST0003 complete [ 8508.906762] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8508.926068] Lustre: Skipped 2 previous similar messages [ 8508.929395] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8508.934835] LustreError: 203715:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8508.935581] LustreError: 203715:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8510.107094] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8518.729855] Lustre: DEBUG MARKER: START OST3 again [ 8525.631211] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8525.873668] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8525.885267] Lustre: Skipped 1 previous similar message [ 8525.897682] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8527.649456] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8527.717958] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8527.718994] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8531.964523] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8555.100778] Lustre: Failing over lustre-OST0003 [ 8555.162390] Lustre: server umount lustre-OST0003 complete [ 8556.532381] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8556.541041] LustreError: 208436:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8556.546330] Lustre: Skipped 1 previous similar message [ 8556.580045] LustreError: 208436:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 8577.023123] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8577.028840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8577.031788] Lustre: Skipped 2 previous similar messages [ 8577.053514] Lustre: Skipped 3 previous similar messages [ 8580.569660] Lustre: server umount lustre-MDT0000 complete [ 8584.486218] LustreError: 202964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788866356 with bad export cookie 9264879966060419382 [ 8584.499925] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8584.505749] LustreError: 202964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8587.238536] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8587.250847] Lustre: Skipped 1 previous similar message [ 8591.156695] Lustre: server umount lustre-MDT0001 complete [ 8595.082283] Lustre: server umount lustre-OST0000 complete [ 8599.371980] Lustre: server umount lustre-OST0001 complete [ 8607.185543] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 8617.510516] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 8660.598832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8660.797669] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8660.821151] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8660.924925] Lustre: lustre-MDT0000: new disk, initializing [ 8661.082557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8661.137553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8665.145146] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8677.262824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8677.557205] Lustre: 212017:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8677.619735] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8677.623659] Lustre: Skipped 1 previous similar message [ 8677.710267] Lustre: lustre-MDT0001: new disk, initializing [ 8677.942820] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8677.952292] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8684.235945] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8693.072029] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8699.281704] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8707.555723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8707.916328] Lustre: 213988:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8709.366557] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8709.556851] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8717.291042] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8728.393203] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8732.476076] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8732.890334] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8738.529207] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8738.739362] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8745.442951] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8745.445848] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8745.454209] LustreError: Skipped 1 previous similar message [ 8745.461456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8745.461468] Lustre: Skipped 2 previous similar messages [ 8745.505156] Lustre: Skipped 4 previous similar messages [ 8755.169375] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8755.522199] Lustre: server umount lustre-OST0000 complete [ 8759.778199] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8765.942983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8765.952291] Lustre: Skipped 7 previous similar messages [ 8766.294967] Lustre: server umount lustre-MDT0000 complete [ 8770.348976] LustreError: 212010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788866542 with bad export cookie 9264879966060421678 [ 8770.356045] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8770.365748] LustreError: 212010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8770.703739] Lustre: server umount lustre-MDT0001 complete [ 8774.739415] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 07:22:26 (1788866546) [ 8791.246320] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 8802.297580] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 8840.076052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8840.311437] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8840.332205] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8840.333933] Lustre: Skipped 1 previous similar message [ 8840.443638] Lustre: lustre-MDT0000: new disk, initializing [ 8840.445528] Lustre: Skipped 1 previous similar message [ 8840.558477] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8840.563807] Lustre: Skipped 1 previous similar message [ 8844.320242] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8854.331254] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8854.516650] Lustre: 218602:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8855.149248] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8860.776540] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8871.071786] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8876.559826] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8884.001773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8884.349805] Lustre: 220572:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8886.256875] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8891.733170] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8903.348798] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8908.676721] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8908.945440] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8914.202850] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8914.475226] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8916.962690] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8916.967865] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8916.979426] LustreError: Skipped 1 previous similar message [ 8916.980954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8916.987890] Lustre: Skipped 3 previous similar messages [ 8917.011739] Lustre: Skipped 1 previous similar message [ 8931.296712] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8931.817580] Lustre: server umount lustre-OST0000 complete [ 8936.929905] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8942.413461] Lustre: server umount lustre-MDT0000 complete [ 8942.564391] LustreError: 219468:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8942.581852] LustreError: 219468:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 17 previous similar messages [ 8945.961445] LustreError: 218591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788866718 with bad export cookie 9264879966060422560 [ 8945.964173] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8945.968879] LustreError: 218591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8946.372824] Lustre: server umount lustre-MDT0001 complete [ 8956.937200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8957.670631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8957.676656] Lustre: Skipped 5 previous similar messages [ 8963.146386] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8973.443467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8979.324381] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8988.105718] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8993.857128] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9001.034753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9008.333529] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 9016.849676] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9028.067943] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9028.070678] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9028.075308] Lustre: Skipped 2 previous similar messages [ 9028.078486] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9028.081183] Lustre: Skipped 7 previous similar messages [ 9032.744642] Lustre: server umount lustre-OST0000 complete [ 9041.905040] Lustre: server umount lustre-MDT0000 complete [ 9045.740311] LustreError: 222772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788866817 with bad export cookie 9264879966060423442 [ 9045.742881] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9045.748265] LustreError: 222772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9046.061962] Lustre: server umount lustre-MDT0001 complete [ 9157.474942] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 9161.179877] Key type lgssc unregistered [ 9161.473525] LNet: 226451:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9161.479471] LNetError: 226451:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9161.497862] LNet: Removed LNI 192.168.202.149@tcp [ 9162.266157] Key type .llcrypt unregistered [ 9162.267322] Key type ._llcrypt unregistered [ 9186.818594] Key type ._llcrypt registered [ 9186.822797] Key type .llcrypt registered [ 9187.000217] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 9204.562855] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 9205.730716] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9205.805390] alg: No test for adler32 (adler32-zlib) [ 9207.080993] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 9207.319210] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [ 9208.970665] Key type lgssc registered [ 9210.316954] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9248.246519] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9248.272269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9249.629723] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9249.651889] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9249.743500] Lustre: lustre-MDT0000: new disk, initializing [ 9249.816237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9249.837942] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9255.269920] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 9267.291189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9267.471735] Lustre: 229803:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9267.532864] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9267.541809] Lustre: Skipped 1 previous similar message [ 9267.650563] Lustre: lustre-MDT0001: new disk, initializing [ 9267.735944] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9267.756855] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9267.770857] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9273.572500] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 9284.486367] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9291.737711] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9300.963180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9301.523673] Lustre: lustre-OST0000: new disk, initializing [ 9301.532255] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9301.545826] Lustre: 231771:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9301.663807] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9309.891548] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 9311.898094] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9311.917561] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9312.135527] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9322.863137] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9329.313691] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9329.621361] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9335.598297] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9335.903192] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9342.951583] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9342.958773] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9342.972208] LustreError: Skipped 1 previous similar message [ 9342.976371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9342.994568] Lustre: Skipped 1 previous similar message [ 9344.683087] Lustre: server umount lustre-OST0000 complete [ 9349.605633] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9349.616148] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9349.636977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9349.645056] Lustre: Skipped 1 previous similar message [ 9353.200697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9353.208062] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9354.870776] Lustre: server umount lustre-MDT0000 complete [ 9359.328779] LustreError: 229794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788867131 with bad export cookie 3358385446699288010 [ 9359.332761] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9359.347646] LustreError: 229794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9360.113363] Lustre: server umount lustre-MDT0001 complete [ 9382.987815] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 9395.606359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9396.112871] LustreError: 234707:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9396.152187] LustreError: 234707:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9396.183247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9401.315908] LustreError: 234708:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9401.991556] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 9406.449259] LustreError: 234707:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9411.567537] LustreError: 234708:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9412.319269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9413.021934] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9419.146791] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 9423.302268] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9431.161239] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9431.724677] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9439.227375] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 9449.195892] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9449.456770] Lustre: lustre-OST0001: new disk, initializing [ 9449.461566] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9449.466531] Lustre: 237018:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9449.626726] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9458.044679] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 9459.412708] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9459.434143] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9459.630606] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9472.189101] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9480.279514] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9494.676799] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 07:34:26 (1788867266) [ 9500.129436] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9500.142551] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9500.169424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9500.178492] Lustre: Skipped 1 previous similar message [ 9505.763156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9505.778532] Lustre: Skipped 5 previous similar messages [ 9511.393612] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9511.908348] Lustre: server umount lustre-MDT0000 complete [ 9515.896131] LustreError: 234689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788867288 with bad export cookie 3358385446699288892 [ 9515.902142] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9515.917106] LustreError: 234689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9516.020950] LustreError: 234704:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9516.033423] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9516.051349] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9516.051358] Lustre: Skipped 5 previous similar messages [ 9516.061446] LustreError: 234704:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9516.125815] Lustre: Skipped 4 previous similar messages [ 9516.956288] Lustre: server umount lustre-MDT0001 complete [ 9524.420412] Lustre: server umount lustre-OST0000 complete [ 9530.696774] Lustre: server umount lustre-OST0001 complete [ 9539.618330] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 9548.986234] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 9590.014983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9590.315809] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9590.384160] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9590.533971] Lustre: lustre-MDT0000: new disk, initializing [ 9590.614290] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9590.643433] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9596.749800] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 9609.394919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9609.696749] Lustre: 240965:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9609.872652] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9609.884128] Lustre: Skipped 1 previous similar message [ 9610.092349] Lustre: lustre-MDT0001: new disk, initializing [ 9610.333542] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9610.347677] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9616.463547] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 9626.983929] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9632.905451] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9640.136257] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9640.445350] Lustre: lustre-OST0000: new disk, initializing [ 9640.449712] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9640.457575] Lustre: 242936:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9641.917077] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9641.929166] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9642.036848] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9648.303415] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 9658.181974] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9663.695224] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9664.089737] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9669.569224] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9669.862759] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9672.677118] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9672.690739] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9672.702138] Lustre: Skipped 1 previous similar message [ 9672.710913] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9672.723183] Lustre: Skipped 2 previous similar messages [ 9678.249454] Lustre: server umount lustre-OST0000 complete [ 9688.924608] Lustre: server umount lustre-MDT0000 complete [ 9693.154131] LustreError: 240978:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9693.178993] LustreError: 240978:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9693.512047] LustreError: 240959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788867465 with bad export cookie 3358385446699291034 [ 9693.519252] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9693.532577] LustreError: 240959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9693.917929] Lustre: server umount lustre-MDT0001 complete [ 9703.843565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9704.704380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9704.707794] Lustre: Skipped 2 previous similar messages [ 9710.054598] LustreError: 245152:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9710.070658] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 9710.072663] LustreError: 245152:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9719.524799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9725.085576] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 9732.195988] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9739.509780] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9748.335342] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9756.210928] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 9763.095758] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9913.835222] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9913.842444] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9913.851740] Lustre: Skipped 2 previous similar messages [ 9913.858575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9913.864408] Lustre: Skipped 6 previous similar messages [ 9916.821531] Lustre: server umount lustre-OST0000 complete [ 9927.614835] Lustre: server umount lustre-MDT0000 complete [ 9929.190320] LustreError: 245148:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9929.211744] LustreError: 245148:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9932.293456] LustreError: 245132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788867704 with bad export cookie 3358385446699291916 [ 9932.312464] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9932.836749] Lustre: server umount lustre-MDT0001 complete [10035.837671] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [10039.369180] Key type lgssc unregistered [10039.761663] LNet: 248760:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10039.772419] LNetError: 248760:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10039.788058] LNet: Removed LNI 192.168.202.149@tcp [10040.923234] Key type .llcrypt unregistered [10040.929495] Key type ._llcrypt unregistered [10070.580284] Key type ._llcrypt registered [10070.585049] Key type .llcrypt registered [10070.729304] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [10088.929509] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [10090.182907] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10090.547477] alg: No test for adler32 (adler32-zlib) [10091.762094] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [10092.052804] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [10093.768194] Key type lgssc registered [10095.208369] Lustre: Echo OBD driver; http://www.lustre.org/ [10145.641419] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [10158.063550] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10158.101947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10159.419991] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10159.474986] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10159.564995] Lustre: lustre-MDT0000: new disk, initializing [10159.748372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10159.785438] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10165.365800] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10177.153570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10177.259867] Lustre: 252826:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10177.305857] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10177.308258] Lustre: Skipped 1 previous similar message [10177.364176] Lustre: lustre-MDT0001: new disk, initializing [10177.440214] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10177.478950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10177.482086] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10180.990539] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10186.122183] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10194.593308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10194.872865] Lustre: lustre-OST0000: new disk, initializing [10194.880967] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10194.889800] Lustre: 254458:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10194.989817] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10201.804992] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10202.170772] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10202.177929] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10202.291402] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10214.711885] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10214.959993] Lustre: lustre-OST0001: new disk, initializing [10214.967549] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10214.979852] Lustre: 255328:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10215.118132] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10222.745388] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10222.767945] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10222.890917] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10223.313409] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10235.454290] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10246.062314] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10258.914704] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10258.919595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10258.939126] Lustre: Skipped 1 previous similar message [10258.955926] Lustre: Skipped 3 previous similar messages [10264.037760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10264.042698] Lustre: Skipped 3 previous similar messages [10264.932505] Lustre: server umount lustre-MDT0000 complete [10269.194423] LustreError: 254469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788868041 with bad export cookie 9937249154813585662 [10269.205481] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10269.212728] LustreError: 254469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10269.830698] Lustre: server umount lustre-MDT0001 complete [10285.008840] Lustre: server umount lustre-OST0000 complete [10299.397609] Lustre: server umount lustre-OST0001 complete [10315.204738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10325.698343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10334.558621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10343.957166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10366.281144] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [10377.595784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10377.614342] Lustre: lustre-MDT0000: reset Object Index mappings [10378.102724] LustreError: 260345:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10378.125278] LustreError: 260345:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10378.173146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10382.962286] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10383.330664] LustreError: 260346:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10388.457141] LustreError: 260345:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10392.494321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10392.507872] Lustre: lustre-MDT0001: reset Object Index mappings [10393.028855] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10398.539177] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10401.772095] Lustre: 261488:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10409.546425] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10410.151958] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10417.721125] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10418.348304] LustreError: 261840:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10423.474981] LustreError: 261843:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10423.494312] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10423.503979] LustreError: 261843:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10427.420035] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10433.021369] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10434.471494] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10443.452752] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10446.863100] Lustre: 263357:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10453.476992] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10453.481505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10453.495291] Lustre: Skipped 3 previous similar messages [10454.524546] Lustre: server umount lustre-MDT0000 complete [10458.360907] LustreError: 260326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788868230 with bad export cookie 9937249154813592459 [10458.370221] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10458.374447] LustreError: 260326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10458.593625] LustreError: 262111:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10458.602552] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10458.613603] Lustre: Skipped 2 previous similar messages [10458.726233] Lustre: server umount lustre-MDT0001 complete [10464.864337] Lustre: server umount lustre-OST0000 complete [10469.301656] Lustre: server umount lustre-OST0001 complete [10491.643939] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 07:51:02 (1788868262) [10509.781995] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [10521.196740] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [10564.458403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10564.695843] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10564.723696] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10564.821937] Lustre: lustre-MDT0000: new disk, initializing [10564.902414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10564.906873] Lustre: Skipped 1 previous similar message [10564.919881] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10569.783596] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10580.973890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10581.116231] Lustre: 267612:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10581.172405] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10581.177516] Lustre: Skipped 1 previous similar message [10581.261783] Lustre: lustre-MDT0001: new disk, initializing [10581.456991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10581.478850] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10586.920314] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10598.053406] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10604.737611] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10612.522993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10612.884824] Lustre: lustre-OST0000: new disk, initializing [10612.888193] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10612.897678] Lustre: 269580:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10614.249877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10614.272870] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10614.425806] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10620.905713] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10631.388483] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10636.978395] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10637.345402] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10643.006743] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10643.372400] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10650.100322] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10650.111719] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10650.119936] Lustre: Skipped 1 previous similar message [10650.123264] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10650.123274] Lustre: Skipped 3 previous similar messages [10651.857543] Lustre: server umount lustre-OST0000 complete [10658.273569] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10658.279063] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10658.292166] Lustre: Skipped 1 previous similar message [10658.300492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10658.316299] Lustre: Skipped 1 previous similar message [10662.559337] Lustre: server umount lustre-MDT0000 complete [10665.449248] LustreError: 267623:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10665.470634] LustreError: 267623:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10666.842338] LustreError: 267604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788868439 with bad export cookie 9937249154813594545 [10666.862310] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10667.289516] Lustre: server umount lustre-MDT0001 complete [10685.408238] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [10695.111040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10695.655422] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10695.662167] Lustre: Skipped 2 previous similar messages [10700.546513] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10700.770357] LustreError: 272516:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10700.777316] LustreError: 272516:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10708.924962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10714.182649] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10718.274808] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10726.224881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10734.278280] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10744.057527] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10744.248863] Lustre: lustre-OST0001: new disk, initializing [10744.253524] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10744.260314] Lustre: 274824:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10745.382058] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10745.399258] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10745.492660] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10752.049061] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10761.599166] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10766.024174] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10771.398681] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10771.534055] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10781.153674] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10781.165845] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10781.183530] Lustre: Skipped 1 previous similar message [10781.191065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10781.198233] Lustre: Skipped 2 previous similar messages [10785.591833] Lustre: server umount lustre-MDT0000 complete [10786.786516] LustreError: 272510:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10786.812845] LustreError: 272510:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10790.587834] LustreError: 273249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788868562 with bad export cookie 9937249154813595434 [10790.610976] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10790.617809] LustreError: 273249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [10791.087039] Lustre: server umount lustre-MDT0001 complete [10806.575187] Lustre: server umount lustre-OST0000 complete [10808.353551] Lustre: 250334:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788868564/real 1788868564] req@ffff968103d14a80 x1875763909605760/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788868580 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10808.399161] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10808.414480] Lustre: Skipped 3 previous similar messages [10811.574804] Lustre: server umount lustre-OST0001 complete [10831.335962] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [10844.058532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10844.794036] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10844.803626] Lustre: Skipped 3 previous similar messages [10850.919299] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10860.691883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10865.838178] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10869.251687] Lustre: 279020:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10876.469684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10883.902991] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10889.225936] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10892.746162] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10898.449816] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10900.608471] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10909.336875] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10913.228776] Lustre: 280898:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10919.005739] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10919.020727] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10929.502493] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 07:58:20 (1788868700) [10948.077644] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10948.086966] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10948.096142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10948.099311] Lustre: Skipped 3 previous similar messages [10951.713668] Lustre: server umount lustre-MDT0000 complete [10954.729071] LustreError: 277880:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10954.748735] LustreError: 277880:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [10955.905944] LustreError: 277861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788868728 with bad export cookie 9937249154813602231 [10955.907991] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10955.916392] LustreError: 277861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10962.359726] Lustre: server umount lustre-MDT0001 complete [10981.344170] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10981.723417] Lustre: server umount lustre-OST0000 complete [10986.864767] Lustre: server umount lustre-OST0001 complete