[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 489254697 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003325] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009011] pid_max: default: 32768 minimum: 301 [ 0.010179] LSM: Security Framework initializing [ 0.011067] Yama: becoming mindful. [ 0.012045] SELinux: Initializing. [ 0.014062] *** VALIDATE selinux *** [ 0.023337] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028419] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029143] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030102] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032003] *** VALIDATE tmpfs *** [ 0.033459] *** VALIDATE proc *** [ 0.034229] *** VALIDATE cgroup *** [ 0.035010] *** VALIDATE cgroup2 *** [ 0.037249] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039094] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041028] Spectre V2 : User space: Vulnerable [ 0.042010] Speculative Store Bypass: Vulnerable [ 0.044928] debug: unmapping init [mem 0xffffffffb6e59000-0xffffffffb6e60fff] [ 0.046164] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047684] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048022] ... version: 2 [ 0.049011] ... bit width: 48 [ 0.050010] ... generic registers: 4 [ 0.051018] ... value mask: 0000ffffffffffff [ 0.052014] ... max period: 00007fffffffffff [ 0.053016] ... fixed-purpose events: 3 [ 0.054012] ... event mask: 000000070000000f [ 0.056215] rcu: Hierarchical SRCU implementation. [ 0.058392] smp: Bringing up secondary CPUs ... [ 0.059571] x86: Booting SMP configuration: [ 0.060026] .... node #0, CPUs: #1 #2 #3 [ 0.063515] smp: Brought up 1 node, 4 CPUs [ 0.065015] smpboot: Max logical packages: 1 [ 0.066019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.240407] node 0 deferred pages initialised in 172ms [ 0.244105] devtmpfs: initialized [ 0.245349] x86/mm: Memory block size: 128MB [ 0.249992] gcov: version magic: 0x41383552 [ 0.253364] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.254111] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.256499] pinctrl core: initialized pinctrl subsystem [ 0.259286] [ 0.259906] ************************************************************* [ 0.261014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.263016] ** ** [ 0.264015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.266015] ** ** [ 0.268016] ** This means that this kernel is built to expose internal ** [ 0.270014] ** IOMMU data structures, which may compromise security on ** [ 0.271013] ** your system. ** [ 0.272013] ** ** [ 0.273014] ** If you see this message and you are not debugging the ** [ 0.274014] ** kernel, report this immediately to your vendor! ** [ 0.276017] ** ** [ 0.277015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.279014] ************************************************************* [ 0.281777] NET: Registered protocol family 16 [ 0.283429] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.285086] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.288076] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.291516] cpuidle: using governor menu [ 0.292390] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.296073] PCI: Using configuration type 1 for base access [ 0.297156] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.307082] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.309038] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.313107] cryptd: max_cpu_qlen set to 1000 [ 0.317265] ACPI: Added _OSI(Module Device) [ 0.318000] ACPI: Added _OSI(Processor Device) [ 0.319012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.320019] ACPI: Added _OSI(Processor Aggregator Device) [ 0.326463] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.333618] ACPI: Interpreter enabled [ 0.335088] ACPI: PM: (supports S0 S3 S4 S5) [ 0.337028] ACPI: Using IOAPIC for interrupt routing [ 0.339254] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.343680] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.355167] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.357059] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.360030] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.364116] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.368400] acpiphp: Slot [2] registered [ 0.370209] acpiphp: Slot [5] registered [ 0.372171] acpiphp: Slot [6] registered [ 0.373129] acpiphp: Slot [7] registered [ 0.375170] acpiphp: Slot [8] registered [ 0.376141] acpiphp: Slot [9] registered [ 0.377130] acpiphp: Slot [10] registered [ 0.378195] acpiphp: Slot [3] registered [ 0.380136] acpiphp: Slot [4] registered [ 0.382130] acpiphp: Slot [11] registered [ 0.383102] acpiphp: Slot [12] registered [ 0.384101] acpiphp: Slot [13] registered [ 0.386155] acpiphp: Slot [14] registered [ 0.387119] acpiphp: Slot [15] registered [ 0.389181] acpiphp: Slot [16] registered [ 0.391108] acpiphp: Slot [17] registered [ 0.393131] acpiphp: Slot [18] registered [ 0.394115] acpiphp: Slot [19] registered [ 0.396111] acpiphp: Slot [20] registered [ 0.398144] acpiphp: Slot [21] registered [ 0.399112] acpiphp: Slot [22] registered [ 0.401121] acpiphp: Slot [23] registered [ 0.403064] acpiphp: Slot [24] registered [ 0.404122] acpiphp: Slot [25] registered [ 0.406109] acpiphp: Slot [26] registered [ 0.407222] acpiphp: Slot [27] registered [ 0.409095] acpiphp: Slot [28] registered [ 0.411107] acpiphp: Slot [29] registered [ 0.412111] acpiphp: Slot [30] registered [ 0.414116] acpiphp: Slot [31] registered [ 0.416068] PCI host bridge to bus 0000:00 [ 0.418026] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.421026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.423025] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.426028] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.429025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.432026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.434304] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.438454] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.442451] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.452916] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.457571] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.460018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.463025] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.466026] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.469597] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.473125] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.475058] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.477960] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.483021] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.498021] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.505926] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.511027] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.520020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.525023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.540021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.550343] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.563017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.568019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.597065] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.607338] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.617022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.632020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.652020] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.667218] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.672021] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.678032] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.693023] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.703172] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.708017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.718026] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.731021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.740968] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.748019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.756022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.775021] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.789119] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.790345] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.793331] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.796743] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.799313] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.805046] iommu: Default domain type: Passthrough [ 0.806554] SCSI subsystem initialized [ 0.807173] ACPI: bus type USB registered [ 0.808265] usbcore: registered new interface driver usbfs [ 0.810119] usbcore: registered new interface driver hub [ 0.812211] usbcore: registered new device driver usb [ 0.815227] pps_core: LinuxPPS API ver. 1 registered [ 0.817014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.821144] PTP clock support registered [ 0.824062] EDAC MC: Ver: 3.0.0 [ 0.826152] PCI: Using ACPI for IRQ routing [ 0.829045] NetLabel: Initializing [ 0.830014] NetLabel: domain hash size = 128 [ 0.832014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.833105] NetLabel: unlabeled traffic allowed by default [ 0.836110] vgaarb: loaded [ 0.837284] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.839018] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.845145] clocksource: Switched to clocksource kvm-clock [ 0.957395] VFS: Disk quotas dquot_6.6.0 [ 0.960918] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.963721] *** VALIDATE ramfs *** [ 0.965239] *** VALIDATE hugetlbfs *** [ 0.967031] pnp: PnP ACPI init [ 0.969650] pnp: PnP ACPI: found 6 devices [ 0.992984] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.996697] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.999228] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.001510] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.003935] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.006528] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.009537] NET: Registered protocol family 2 [ 1.012236] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.017322] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.021219] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.026108] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.029362] TCP: Hash tables configured (established 65536 bind 65536) [ 1.032527] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.035812] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.038704] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.062703] NET: Registered protocol family 1 [ 1.067764] RPC: Registered named UNIX socket transport module. [ 1.070094] RPC: Registered udp transport module. [ 1.072109] RPC: Registered tcp transport module. [ 1.073609] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.076274] NET: Registered protocol family 44 [ 1.077905] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.080051] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.082281] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.084606] PCI: CLS 0 bytes, default 64 [ 1.087125] Unpacking initramfs... [ 2.542234] debug: unmapping init [mem 0xffff9b567cc54000-0xffff9b567ffbffff] [ 2.547415] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.551107] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.555743] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.051030] Initialise system trusted keyrings [ 3.052785] Key type blacklist registered [ 3.054804] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.064140] zbud: loaded [ 3.067271] *** VALIDATE nfs *** [ 3.068747] *** VALIDATE nfs4 *** [ 3.070547] pstore: using deflate compression [ 3.074188] Platform Keyring initialized [ 3.181469] NET: Registered protocol family 38 [ 3.183966] Key type asymmetric registered [ 3.185922] Asymmetric key parser 'x509' registered [ 3.187945] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.191077] io scheduler mq-deadline registered [ 3.192695] io scheduler kyber registered [ 3.194332] io scheduler bfq registered [ 3.195907] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.199159] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.202163] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.205147] ACPI: Power Button [PWRF] [ 3.211846] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.219223] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.237928] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.250225] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.275189] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.308397] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.345442] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.355113] Non-volatile memory driver v1.3 [ 3.357127] Linux agpgart interface v0.103 [ 3.396847] virtio_blk virtio1: [vda] 146216 512-byte logical blocks (74.9 MB/71.4 MiB) [ 3.399857] vda: detected capacity change from 0 to 74862592 [ 3.417725] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.420993] vdb: detected capacity change from 0 to 1073741824 [ 3.441278] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.444453] vdc: detected capacity change from 0 to 2621440000 [ 3.464451] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.467593] vdd: detected capacity change from 0 to 2621440000 [ 3.484512] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.487761] vde: detected capacity change from 0 to 4294967296 [ 3.500836] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.503276] vdf: detected capacity change from 0 to 4294967296 [ 3.508472] libphy: Fixed MDIO Bus: probed [ 3.515975] usbcore: registered new interface driver usbserial_generic [ 3.518826] usbserial: USB Serial support registered for generic [ 3.521759] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.526356] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.528413] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.531163] mousedev: PS/2 mouse device common for all mice [ 3.534615] rtc_cmos 00:05: RTC can wake from S4 [ 3.538248] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.540110] rtc_cmos 00:05: registered as rtc0 [ 3.544551] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.547145] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.548944] intel_pstate: CPU model not supported [ 3.556198] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.560251] hid: raw HID events driver (C) Jiri Kosina [ 3.563461] usbcore: registered new interface driver usbhid [ 3.565597] usbhid: USB HID core driver [ 3.567387] drop_monitor: Initializing network drop monitor service [ 3.569854] Initializing XFRM netlink socket [ 3.572188] NET: Registered protocol family 10 [ 3.576996] Segment Routing with IPv6 [ 3.578656] NET: Registered protocol family 17 [ 3.581361] mpls_gso: MPLS GSO support [ 3.587836] RAS: Correctable Errors collector initialized. [ 3.590484] AVX version of gcm_enc/dec engaged. [ 3.592468] AES CTR mode by8 optimization enabled [ 3.684785] sched_clock: Marking stable (3684759742, 0)->(4618883461, -934123719) [ 3.689389] registered taskstats version 1 [ 3.691701] Loading compiled-in X.509 certificates [ 3.694382] zswap: loaded using pool lzo/zbud [ 3.725750] Key type big_key registered [ 3.742373] Key type encrypted registered [ 3.744358] ima: No TPM chip found, activating TPM-bypass! [ 3.746418] ima: Allocated hash algorithm: sha1 [ 3.748645] ima: No architecture policies found [ 3.750895] evm: Initialising EVM extended attributes: [ 3.753177] evm: security.selinux [ 3.754682] evm: security.ima [ 3.756245] evm: security.capability [ 3.758091] evm: HMAC attrs: 0x1 [ 3.761248] rtc_cmos 00:05: setting system clock to 2026-08-24 08:59:26 UTC (1787561966) [ 3.769416] debug: unmapping init [mem 0xffffffffb7e03000-0xffffffffb7ffffff] [ 3.773213] debug: unmapping init [mem 0xffffffffb6b82000-0xffffffffb6e58fff] [ 3.782080] Write protecting the kernel read-only data: 28672k [ 3.785827] debug: unmapping init [mem 0xffffffffb5203000-0xffffffffb53fffff] [ 3.791169] debug: unmapping init [mem 0xffffffffb5b14000-0xffffffffb5bfffff] [ 3.827600] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.835775] systemd[1]: Detected virtualization kvm. [ 3.837992] systemd[1]: Detected architecture x86-64. [ 3.840099] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.868191] systemd[1]: No hostname configured. [ 3.870773] systemd[1]: Set hostname to . [ 3.873384] random: systemd: uninitialized urandom read (16 bytes read) [ 3.876605] systemd[1]: Initializing machine ID from random generator. [ 3.931272] random: ln: uninitialized urandom read (6 bytes read) [ 4.017463] random: systemd: uninitialized urandom read (16 bytes read) [ 4.020978] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.026806] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 4.035104] systemd[1]: Started Memstrack Anylazing Service. [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Timers. Starting Setup Virtual Console... [ OK ] Reached target Swap. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Paths. Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.699816] device-mapper: uevent: version 1.0.3 [ 4.702128] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.410925] virtio_net virtio0 ens2: renamed from eth0 [ 5.446710] random: fast init done [ 5.475461] scsi host0: ata_piix [ 5.491489] scsi host1: ata_piix [ 5.493248] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.496146] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.770151] dracut-initqueue[581]: RTNETLINK answers: File exists [ 10.270398] random: crng init done [ 10.272133] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 12.023388] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 15.058920] printk: systemd: 25 output lines suppressed due to ratelimiting [ 15.832604] SELinux: Disabled at runtime. [ 15.978217] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 15.991811] systemd[1]: Detected virtualization kvm. [ 15.993729] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 17.427351] systemd[1]: initrd-switch-root.service: Succeeded. [ 17.433868] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 17.444715] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 17.452123] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 17.460694] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 17.485601] systemd[1]: Starting Journal Service... Starting Journal Service... [ 17.506475] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Kernel Socket. Starting Apply Kernel Variables... [ OK ] Reached target Paths. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Starting udev Coldplug all Devices... Starting Remount Root and Kernel File Systems... Mounting Huge Pages File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-getty.slice. [ 17.994665] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice User and Session Slice. Mounting Kernel Debug File System... [ OK ] Reached target Slices. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting POSIX Message Queue File System... [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 19.020687] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 19.850220] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 20.075913] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 20.406845] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 20.483660] EDAC sbridge: Ver: 1.1.2 [ 24.499617] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (7s / no limit)[ 24.869185] NFS: Registering the id_resolver key type [ 24.870726] Key type id_resolver registered [ 24.872630] Key type id_legacy registered [** ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg654-server login: [ 35.265773] hrtimer: interrupt took 5916900 ns [ 80.651814] libcfs: loading out-of-tree module taints kernel. [ 80.725155] Key type ._llcrypt registered [ 80.727221] Key type .llcrypt registered [ 80.799502] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_hostid [ 97.376632] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 99.134598] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 99.163284] alg: No test for adler32 (adler32-zlib) [ 100.800433] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 102.201646] LNet: Added LNI 192.168.206.154@tcp [8/256/0/180] [ 104.168150] Key type lgssc registered [ 106.634892] Lustre: Echo OBD driver; http://www.lustre.org/ [ 121.745206] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 159.608472] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 171.462563] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 171.491924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 172.715587] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 172.741213] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 172.829598] Lustre: lustre-MDT0000: new disk, initializing [ 172.895551] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 172.912882] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 176.927938] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 190.186189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 190.266798] Lustre: 6520:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 190.296996] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 190.300185] Lustre: Skipped 1 previous similar message [ 190.368365] Lustre: lustre-MDT0001: new disk, initializing [ 190.416730] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 190.434706] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 190.448381] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 195.222684] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 200.114590] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 209.129174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 209.410987] Lustre: lustre-OST0000: new disk, initializing [ 209.413936] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 209.419359] Lustre: 8460:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 209.482368] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 214.102922] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 214.121882] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 214.266869] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 216.732093] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 232.160323] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 232.347522] Lustre: lustre-OST0001: new disk, initializing [ 232.360651] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 232.374846] Lustre: 9532:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 232.475883] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 240.336398] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 242.197445] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 242.206347] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 242.271591] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 251.800835] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 259.160939] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 266.479281] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing check_logdir /tmp/testlogs/ [ 272.060818] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing yml_node [ 275.762183] Lustre: DEBUG MARKER: Client: 2.17.57.83 [ 278.039113] Lustre: DEBUG MARKER: MDS: 2.17.57.83 [ 280.435744] Lustre: DEBUG MARKER: OSS: 2.17.57.83 [ 281.853866] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Aug 24 05:04:02 EDT 2026 [ 298.458774] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 300.029438] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 318.443480] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 318.458728] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 318.489523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 322.937571] Lustre: server umount lustre-MDT0000 complete [ 326.554427] LustreError: 6514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787562289 with bad export cookie 18418554427702052815 [ 326.559394] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 326.568711] LustreError: 6514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 329.189683] LustreError: 6529:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 329.189877] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 329.193129] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 329.193138] Lustre: Skipped 3 previous similar messages [ 329.210979] LustreError: 6529:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 329.251252] Lustre: Skipped 4 previous similar messages [ 331.242815] LustreError: 9088:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 331.245630] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 331.256359] LustreError: 9088:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 331.260254] Lustre: Skipped 2 previous similar messages [ 333.060476] Lustre: server umount lustre-MDT0001 complete [ 336.962834] Lustre: server umount lustre-OST0000 complete [ 340.663404] Lustre: server umount lustre-OST0001 complete [ 347.088418] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_hostid [ 356.106053] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 392.829384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 393.242843] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 393.286074] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 393.388450] Lustre: lustre-MDT0000: new disk, initializing [ 393.534342] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 393.562610] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 397.840301] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 408.386192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 408.545139] Lustre: 15084:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 408.616200] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 408.619816] Lustre: Skipped 1 previous similar message [ 408.731251] Lustre: lustre-MDT0001: new disk, initializing [ 408.890568] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 408.939049] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 408.947303] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 414.374904] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 424.155794] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 430.654121] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 437.393807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 437.773703] Lustre: lustre-OST0000: new disk, initializing [ 437.783118] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 437.802972] Lustre: 17052:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 437.957582] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 439.527423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 439.541967] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 439.687235] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 444.870971] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 455.659798] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 462.188464] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 462.527574] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 468.205976] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 468.431347] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 475.619435] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 475.622779] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 475.624908] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 475.657676] Lustre: Skipped 1 previous similar message [ 475.669090] LustreError: Skipped 1 previous similar message [ 480.743369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 480.748911] Lustre: Skipped 2 previous similar messages [ 485.344124] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 485.462359] Lustre: server umount lustre-OST0000 complete [ 490.982149] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 490.982324] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 490.999615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 491.030733] Lustre: Skipped 1 previous similar message [ 495.858765] Lustre: server umount lustre-MDT0000 complete [ 496.098466] LustreError: 15089:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 496.118371] LustreError: 15089:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 499.834669] LustreError: 15074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787562462 with bad export cookie 18418554427702054341 [ 499.844971] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 499.855654] LustreError: 15074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 500.327165] Lustre: server umount lustre-MDT0001 complete [ 505.813224] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 05:07:46 (1787562466) [ 524.266420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 524.727511] LustreError: 19928:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 524.788936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 529.918455] LustreError: 19929:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 530.852191] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 539.677658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 545.049473] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 553.434744] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 559.820931] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 562.342636] Lustre: Failing over lustre-MDT0000 [ 562.634689] Lustre: server umount lustre-MDT0000 complete [ 564.387148] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 564.404064] LustreError: 20657:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 564.404190] Lustre: Skipped 1 previous similar message [ 564.433818] LustreError: 20657:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 581.088362] Lustre: 3658:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787562527/real 1787562527] req@ffff9b56c5ba1180 x1874394678981248/t0(0) o400->MGC192.168.206.154@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787562543 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 581.124137] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 581.143031] LustreError: 19929:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 581.156922] LustreError: 19929:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 581.696597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 590.369396] LustreError: 3654:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9b56eef32300 x1874394678984192/t0(0) o250->MGC192.168.206.154@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 590.848178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 590.855570] Lustre: Skipped 1 previous similar message [ 595.947185] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 595.964258] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 595.995995] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 596.099436] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 606.543146] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 611.299090] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 611.304097] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 611.311387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 611.315604] Lustre: Skipped 1 previous similar message [ 616.516066] Lustre: server umount lustre-MDT0000 complete [ 619.493146] LustreError: 19929:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 619.515063] LustreError: 19929:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 620.286670] LustreError: 19908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787562583 with bad export cookie 18418554427702055762 [ 620.300843] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 620.308381] LustreError: 19908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 620.723137] Lustre: server umount lustre-MDT0001 complete [ 628.826675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 633.739260] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 642.100317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 646.620786] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 653.847412] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 659.462597] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 661.666665] Lustre: Failing over lustre-MDT0000 [ 661.910720] Lustre: server umount lustre-MDT0000 complete [ 663.010842] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 663.023809] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 663.034838] Lustre: Skipped 1 previous similar message [ 680.418095] Lustre: 3655:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787562627/real 1787562627] req@ffff9b55c362bb80 x1874394679011200/t0(0) o400->MGC192.168.206.154@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787562643 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 680.441335] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 681.656468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 685.538509] LustreError: 23282:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 685.568330] LustreError: 23282:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 690.657497] LustreError: 3654:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9b56c5ba3b80 x1874394679014016/t0(0) o250->MGC192.168.206.154@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 691.039519] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 695.081890] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 696.294424] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 696.303680] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 696.331634] Lustre: Skipped 1 previous similar message [ 696.356901] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 704.869301] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 709.562750] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 709.563859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 709.589422] Lustre: Skipped 2 previous similar messages [ 709.609493] Lustre: Skipped 3 previous similar messages [ 715.999916] Lustre: server umount lustre-MDT0000 complete [ 720.331702] LustreError: 23262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787562683 with bad export cookie 18418554427702056700 [ 720.341588] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 720.349123] LustreError: 23262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 720.673330] Lustre: server umount lustre-MDT0001 complete [ 728.355280] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 05:11:28 (1787562688) [ 744.926824] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_hostid [ 754.287916] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 793.518176] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 802.839777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 803.126949] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 803.147397] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 803.230411] Lustre: lustre-MDT0000: new disk, initializing [ 803.355037] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 803.361392] Lustre: Skipped 3 previous similar messages [ 803.375202] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 808.787496] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 818.492091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 818.751419] Lustre: 29654:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 819.084295] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 823.210217] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 831.549552] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 836.022738] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 842.393712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 842.630928] Lustre: 31625:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 844.378109] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 844.464713] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 844.544652] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 844.573042] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 849.430297] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 857.653910] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 862.798212] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 880.612105] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 880.614945] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 880.615307] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 880.615311] Lustre: Skipped 2 previous similar messages [ 880.627366] Lustre: Skipped 1 previous similar message [ 885.758266] Lustre: server umount lustre-OST0000 complete [ 895.971525] LustreError: 29666:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 895.995903] LustreError: 29666:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 896.217879] Lustre: server umount lustre-MDT0000 complete [ 900.134954] LustreError: 29646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787562862 with bad export cookie 18418554427702057106 [ 900.138379] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 900.145065] LustreError: 29646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 900.704893] Lustre: server umount lustre-MDT0001 complete [ 911.202951] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing unload_modules_local [ 913.856444] Key type lgssc unregistered [ 914.142605] LNet: 33607:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 914.150767] LNetError: 33607:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 914.165994] LNet: Removed LNI 192.168.206.154@tcp [ 915.024344] Key type .llcrypt unregistered [ 915.027033] Key type ._llcrypt unregistered [ 939.098632] Key type ._llcrypt registered [ 939.101563] Key type .llcrypt registered [ 939.168709] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 940.134648] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 940.181334] alg: No test for adler32 (adler32-zlib) [ 941.380393] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 941.727582] LNet: Added LNI 192.168.206.154@tcp [8/256/0/180] [ 943.432212] Key type lgssc registered [ 944.929375] Lustre: Echo OBD driver; http://www.lustre.org/ [ 952.241364] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 952.259506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 953.608217] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 957.875733] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 964.069501] LustreError: 35024:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 964.088872] LustreError: 35024:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 966.424608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 966.833734] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 970.638857] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 976.965416] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 981.497411] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 987.573076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 987.878117] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 991.990776] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 994.974043] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 999.183754] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1002.416315] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1015.804470] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 05:16:16 (1787562976) [ 1017.697092] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1019.411910] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1028.682107] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 05:16:27 (1787562987) [ 1038.404774] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 05:16:38 (1787562998) [ 1039.440226] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1049.277261] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1049.288526] Lustre: Skipped 2 previous similar messages [ 1051.231292] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 05:16:51 (1787563011) [ 1054.894650] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1054.900039] Lustre: Skipped 3 previous similar messages [ 1072.370507] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1072.377775] Lustre: Skipped 6 previous similar messages [ 1075.096652] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 05:17:15 (1787563035) [ 1078.010735] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1081.585217] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1088.762032] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 05:17:29 (1787563049) [ 1090.786695] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1090.794462] Lustre: Skipped 3 previous similar messages [ 1103.454382] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 05:17:43 (1787563063) [ 1120.228833] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1120.236646] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1120.238795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1120.257105] Lustre: Skipped 1 previous similar message [ 1125.347657] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1125.362229] Lustre: Skipped 2 previous similar messages [ 1130.474073] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1130.479526] Lustre: Skipped 1 previous similar message [ 1132.000130] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1132.214841] Lustre: server umount lustre-OST0000 complete [ 1135.585684] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1135.595844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1135.596156] Lustre: Skipped 1 previous similar message [ 1135.616371] Lustre: Skipped 1 previous similar message [ 1140.707233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1140.712218] Lustre: Skipped 1 previous similar message [ 1143.780746] Lustre: server umount lustre-MDT0000 complete [ 1145.825994] LustreError: 35024:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1145.842048] LustreError: 35024:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1147.402513] LustreError: 35004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787563110 with bad export cookie 1881527076738647402 [ 1147.407647] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1147.419827] LustreError: 35004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1147.781747] Lustre: server umount lustre-MDT0001 complete [ 1158.694681] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing unload_modules_local [ 1161.299097] Key type lgssc unregistered [ 1161.614813] LNet: 42882:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1161.623108] LNetError: 42882:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1161.640958] LNet: Removed LNI 192.168.206.154@tcp [ 1162.377534] Key type .llcrypt unregistered [ 1162.380132] Key type ._llcrypt unregistered [ 1177.800782] Key type ._llcrypt registered [ 1177.807111] Key type .llcrypt registered [ 1177.919474] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_hostid [ 1194.443734] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 1195.893656] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1196.135106] alg: No test for adler32 (adler32-zlib) [ 1197.220904] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 1197.556658] LNet: Added LNI 192.168.206.154@tcp [8/256/0/180] [ 1199.320173] Key type lgssc registered [ 1200.640733] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1233.366680] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1233.394900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1234.555363] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1234.572478] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1234.643476] Lustre: lustre-MDT0000: new disk, initializing [ 1234.737444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1234.748780] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1239.201263] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 1249.613448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1249.795547] Lustre: 46035:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1249.866751] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1249.870362] Lustre: Skipped 1 previous similar message [ 1249.928446] Lustre: lustre-MDT0001: new disk, initializing [ 1250.028449] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1250.065189] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1250.075734] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1254.379771] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 1261.780873] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1265.874210] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1271.080208] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1271.319331] Lustre: lustre-OST0000: new disk, initializing [ 1271.323449] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1271.329438] Lustre: 48003:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1271.415886] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1277.462453] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 1278.585089] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1278.597159] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1278.784124] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1286.897321] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1291.358061] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1291.570872] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1295.370951] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1295.582250] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1299.429519] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1299.430718] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1299.438607] Lustre: Skipped 1 previous similar message [ 1303.416261] Lustre: server umount lustre-OST0000 complete [ 1309.667578] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1309.672379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1309.677182] Lustre: Skipped 1 previous similar message [ 1309.683236] Lustre: Skipped 2 previous similar messages [ 1313.287272] Lustre: server umount lustre-MDT0000 complete [ 1317.452507] LustreError: 46027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787563280 with bad export cookie 7035894767924258896 [ 1317.464133] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1317.467876] LustreError: 46027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1318.528502] Lustre: server umount lustre-MDT0001 complete [ 1327.653753] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 05:21:27 (1787563287) [ 1335.399865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1336.168564] LustreError: 50461:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1336.187347] LustreError: 50461:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1336.273066] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1341.359689] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 1341.415851] LustreError: 50462:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1345.520639] LustreError: 50461:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1349.501980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1349.907687] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1354.320892] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 1361.084986] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1366.278790] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1373.183598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1373.538034] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1379.963632] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 1386.256768] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1391.372938] LustreError: 52988:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1392.172378] Lustre: Modifying parameter general.timeout=1 in log params [ 1393.804270] Lustre: Modifying parameter general.timeout=3 in log params [ 1393.816522] Lustre: Skipped 1 previous similar message [ 1396.442635] Lustre: Modifying parameter general.timeout=6 in log params [ 1396.446658] Lustre: Skipped 2 previous similar messages [ 1400.444512] Lustre: Modifying parameter general.timeout=11 in log params [ 1400.454510] Lustre: Skipped 4 previous similar messages [ 1409.318042] Lustre: Modifying parameter general.timeout=22 in log params [ 1409.325236] Lustre: Skipped 10 previous similar messages [ 1426.199093] Lustre: Modifying parameter general.timeout=37 in log params [ 1426.201773] Lustre: Skipped 14 previous similar messages [ 1459.011797] Lustre: Modifying parameter general.timeout=69 in log params [ 1459.023944] Lustre: Skipped 31 previous similar messages [ 1519.622799] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 05:24:39 (1787563479) [ 1522.533466] Lustre: 59924:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1523.456521] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 in log params [ 1523.471099] Lustre: Skipped 63 previous similar messages [ 1524.502367] Lustre: 60022:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1528.165347] Lustre: 60236:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1528.174633] Lustre: 60236:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1557.600937] Lustre: 61793:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1557.610457] Lustre: 61793:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1557.630750] LustreError: 61793:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1561.447111] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 05:25:21 (1787563521) [ 1575.397031] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1575.408245] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1575.416116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1579.601479] Lustre: server umount lustre-MDT0000 complete [ 1581.028306] LustreError: 50461:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1581.042724] LustreError: 50461:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1583.163523] LustreError: 50444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787563545 with bad export cookie 7035894767924259778 [ 1583.166839] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1583.175450] LustreError: 50444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1583.663023] Lustre: server umount lustre-MDT0001 complete [ 1598.342735] Lustre: server umount lustre-OST0000 complete [ 1624.422705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1624.775653] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1625.084629] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1629.724779] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 1638.399249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1638.545400] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1638.586039] Lustre: 64325:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1644.230215] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 1651.141166] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1656.465179] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1662.390058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1662.637533] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1662.941062] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1662.947762] Lustre: Skipped 1 previous similar message [ 1669.539987] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 1677.339619] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1679.231084] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1679.237522] Lustre: Skipped 20 previous similar messages [ 1679.294339] Lustre: 66890:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1690.595979] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1690.602462] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1690.617402] Lustre: Skipped 2 previous similar messages [ 1690.625515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1690.634411] Lustre: Skipped 2 previous similar messages [ 1693.151595] Lustre: server umount lustre-OST0000 complete [ 1700.322113] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1700.331811] LustreError: Skipped 1 previous similar message [ 1700.338565] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1700.364829] Lustre: Skipped 1 previous similar message [ 1700.372036] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1700.377989] Lustre: Skipped 2 previous similar messages [ 1702.803825] Lustre: server umount lustre-MDT0000 complete [ 1706.465927] LustreError: 65191:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1706.492115] LustreError: 65191:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1706.716843] LustreError: 64318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787563669 with bad export cookie 7035894767924262781 [ 1706.730896] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1707.011461] Lustre: server umount lustre-MDT0001 complete [ 1719.201242] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing unload_modules_local [ 1723.322704] Key type lgssc unregistered [ 1723.961727] LNet: 68107:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1723.976778] LNetError: 68107:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1724.017405] LNet: Removed LNI 192.168.206.154@tcp [ 1725.446239] Key type .llcrypt unregistered [ 1725.450322] Key type ._llcrypt unregistered [ 1753.085207] Key type ._llcrypt registered [ 1753.091980] Key type .llcrypt registered [ 1753.197783] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 1754.192769] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1754.223141] alg: No test for adler32 (adler32-zlib) [ 1755.336530] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 1755.676958] LNet: Added LNI 192.168.206.154@tcp [8/256/0/180] [ 1757.410421] Key type lgssc registered [ 1758.522975] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1766.542273] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1766.568333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1768.091437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1772.902719] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 1777.641573] LustreError: 69522:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1777.676618] LustreError: 69522:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1782.595913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1782.755414] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1783.017064] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1786.897465] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 1793.403925] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1798.805887] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1804.889717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1805.477367] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1811.362841] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 1818.087153] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1826.056344] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1833.442961] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1833.455196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1833.465219] Lustre: Skipped 2 previous similar messages [ 1833.472892] Lustre: Skipped 2 previous similar messages [ 1838.563057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1838.572569] Lustre: Skipped 2 previous similar messages [ 1839.760895] Lustre: server umount lustre-MDT0000 complete [ 1843.502518] LustreError: 69502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787563806 with bad export cookie 11575328690468361535 [ 1843.511057] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1843.515596] LustreError: 69502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1843.698775] LustreError: 69518:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1843.712625] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1843.840168] Lustre: server umount lustre-MDT0001 complete [ 1850.381314] Lustre: server umount lustre-OST0000 complete [ 1876.598793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1876.910252] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1876.942620] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1877.337214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1881.690328] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 1890.083556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1890.185199] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1890.211951] Lustre: 74394:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1890.670743] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1895.236970] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 1903.429175] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1909.366430] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1915.314234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1915.516821] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1915.872258] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1921.351384] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 1928.109498] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1929.592844] Lustre: 76970:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1931.426700] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1937.381820] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1937.384616] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1937.389813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1937.398193] LustreError: Skipped 1 previous similar message [ 1937.439892] Lustre: Skipped 1 previous similar message [ 1942.564545] Lustre: server umount lustre-OST0000 complete [ 1947.108955] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1947.116843] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1947.128706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1947.142232] Lustre: Skipped 1 previous similar message [ 1952.278818] Lustre: server umount lustre-MDT0000 complete [ 1952.742017] LustreError: 74401:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1952.761206] LustreError: 74401:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1956.158804] LustreError: 75139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787563918 with bad export cookie 11575328690468362382 [ 1956.164676] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1956.175799] LustreError: 75139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1956.534375] Lustre: server umount lustre-MDT0001 complete [ 1968.141622] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing unload_modules_local [ 1971.450253] Key type lgssc unregistered [ 1971.804709] LNet: 78181:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1971.815670] LNetError: 78181:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1972.844886] LNet: Removed LNI 192.168.206.154@tcp [ 1973.633841] Key type .llcrypt unregistered [ 1973.636167] Key type ._llcrypt unregistered [ 1999.533801] Key type ._llcrypt registered [ 1999.540407] Key type .llcrypt registered [ 1999.713126] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 2001.235660] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2001.298316] alg: No test for adler32 (adler32-zlib) [ 2002.448500] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 2002.697751] LNet: Added LNI 192.168.206.154@tcp [8/256/0/180] [ 2004.416300] Key type lgssc registered [ 2005.739187] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2013.237650] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2013.259870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2014.817087] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2019.579871] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 2024.937812] LustreError: 79597:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2024.954294] LustreError: 79597:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2028.142667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2028.818243] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2033.699855] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 2040.813829] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2045.712558] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2052.416949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2053.217952] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2059.030804] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 2066.824263] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2072.830074] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2073.779546] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2075.545261] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2075.551911] Lustre: Skipped 1 previous similar message [ 2077.989324] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2077.994286] Lustre: Skipped 2 previous similar messages [ 2082.607772] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2082.611829] Lustre: Skipped 5 previous similar messages [ 2091.306829] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2091.314963] Lustre: Skipped 8 previous similar messages [ 2107.873884] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2107.880776] Lustre: Skipped 15 previous similar messages [ 2140.418151] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2140.424579] Lustre: Skipped 36 previous similar messages [ 2170.161489] Lustre: 87695:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2172.749174] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 05:35:33 (1787564133) [ 2174.279360] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2176.052601] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 05:35:36 (1787564136) [ 2182.625248] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2182.632054] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2182.645250] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2182.668161] Lustre: Skipped 1 previous similar message [ 2185.873912] Lustre: server umount lustre-OST0000 complete [ 2192.354774] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2192.363906] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2192.381169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2192.393106] Lustre: Skipped 1 previous similar message [ 2195.823787] Lustre: server umount lustre-MDT0000 complete [ 2197.989142] LustreError: 80328:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2198.019667] LustreError: 80328:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2199.016277] LustreError: 80339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787564161 with bad export cookie 15181460314614657496 [ 2199.028920] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2199.034390] LustreError: 80339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2199.334907] Lustre: server umount lustre-MDT0001 complete [ 2209.737803] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing unload_modules_local [ 2212.549572] Key type lgssc unregistered [ 2212.896157] LNet: 88945:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2212.906322] LNetError: 88945:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2212.935718] LNet: Removed LNI 192.168.206.154@tcp [ 2213.816672] Key type .llcrypt unregistered [ 2213.822106] Key type ._llcrypt unregistered [ 2218.334850] Key type ._llcrypt registered [ 2218.338305] Key type .llcrypt registered [ 2218.451966] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2223.063601] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules [ 2223.989122] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2224.018970] alg: No test for adler32 (adler32-zlib) [ 2225.187089] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 2225.513920] LNet: Added LNI 192.168.206.154@tcp [8/256/0/180] [ 2227.232189] Key type lgssc registered [ 2228.721310] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2229.708047] LustreError: 90151:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2245.721831] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 2251.342067] LustreError: 90151:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2255.067358] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2255.082992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2256.650174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2261.199567] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 2267.108047] LustreError: 91175:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2267.133248] LustreError: 91175:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2268.760660] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 05:37:09 (1787564229) [ 2272.226667] LustreError: 91174:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2272.641439] LustreError: 91206:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2272.659555] LustreError: 91206:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2272.672274] LustreError: 91206:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 0, failed: rc = -5 [ 2272.985694] Lustre: server umount lustre-MDT0000 complete [ 2286.577677] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing unload_modules_local [ 2289.134808] Key type lgssc unregistered [ 2289.370792] LNet: 92711:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2289.381339] LNetError: 92711:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2290.406456] LNet: Removed LNI 192.168.206.154@tcp [ 2291.103953] Key type .llcrypt unregistered [ 2291.107872] Key type ._llcrypt unregistered [ 2310.210978] Key type ._llcrypt registered [ 2310.213490] Key type .llcrypt registered [ 2310.353041] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 2311.139547] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2311.305092] alg: No test for adler32 (adler32-zlib) [ 2312.298256] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 2312.532122] LNet: Added LNI 192.168.206.154@tcp [8/256/0/180] [ 2314.176185] Key type lgssc registered [ 2315.429831] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2323.352594] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2323.372986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2324.946398] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2329.930159] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 2335.212496] LustreError: 93928:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2335.227683] LustreError: 93928:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2339.355240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2340.140223] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2344.145519] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 2350.905235] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2356.128904] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2362.687375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2363.556271] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2369.404532] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 2376.017394] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2419.774308] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 05:39:40 (1787564380) [ 2421.238941] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2422.697648] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 05:39:43 (1787564383) [ 2426.856041] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2426.865240] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2426.887956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2431.463580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2431.471927] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2433.103547] Lustre: server umount lustre-MDT0000 complete [ 2436.446817] LustreError: 93910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787564399 with bad export cookie 3344887662662324455 [ 2436.450835] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2436.472888] LustreError: 93910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2436.578935] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2436.580545] LustreError: 93923:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2436.606848] Lustre: Skipped 1 previous similar message [ 2436.607799] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2436.607815] Lustre: Skipped 2 previous similar messages [ 2436.821917] Lustre: server umount lustre-MDT0001 complete [ 2442.843849] Lustre: server umount lustre-OST0000 complete [ 2454.123078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2454.497306] LustreError: 98199:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2454.781953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2458.800049] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 2460.130941] LustreError: 98200:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2465.254903] LustreError: 98199:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2465.957761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2466.527506] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2471.184733] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 2478.103873] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2483.296286] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2494.981323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2495.214260] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2495.217633] LustreError: 98186:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2495.265063] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2495.274962] LustreError: 100283:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2495.284637] LustreError: 100283:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2495.291137] LustreError: 100283:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2495.445322] Lustre: server umount lustre-OST0000 complete [ 2495.447325] LustreError: 100283:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2504.275609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2504.484832] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2504.490689] LustreError: 98186:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2504.510399] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2504.515855] LustreError: 100749:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2504.519657] LustreError: 100749:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2504.522943] LustreError: 100749:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2504.569816] Lustre: server umount lustre-OST0000 complete [ 2504.573070] LustreError: 100749:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2513.242389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2513.435398] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2513.441991] Lustre: Found index 0 for lustre-OST0000, updating log [ 2513.448102] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2513.543371] Lustre: lustre-OST0000: new disk, initializing [ 2513.549819] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2513.560633] Lustre: 101282:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2513.788392] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2519.295018] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 2522.245471] Lustre: Failing over lustre-OST0000 [ 2522.353306] Lustre: server umount lustre-OST0000 complete [ 2527.714979] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2527.720323] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2527.734271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2531.453741] Lustre: server umount lustre-MDT0000 complete [ 2534.356804] LustreError: 98179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787564497 with bad export cookie 3344887662662327199 [ 2534.358978] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2534.367907] LustreError: 98179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2534.707872] Lustre: server umount lustre-MDT0001 complete [ 2541.944071] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 05:41:42 (1787564502) [ 2556.857543] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 2567.077333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2567.479524] LustreError: 103626:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2567.622157] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2572.897354] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 2581.729107] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2587.081678] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 2590.220116] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2597.379830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2605.509407] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 2614.259254] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2614.347863] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2614.430811] Lustre: lustre-OST0001: new disk, initializing [ 2614.435788] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2614.442755] Lustre: 105943:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2614.789137] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2614.796026] Lustre: Skipped 2 previous similar messages [ 2621.009195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2621.020957] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2621.161216] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2621.225841] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 2628.481333] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2634.151928] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2638.822177] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2638.831493] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2638.853601] Lustre: Skipped 1 previous similar message [ 2638.873306] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2638.881889] Lustre: Skipped 2 previous similar messages [ 2644.811740] Lustre: server umount lustre-MDT0001 complete [ 2647.016478] LustreError: 103622:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2647.037829] LustreError: 103622:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2653.291909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2653.406995] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2653.415891] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2653.470501] Lustre: 103615:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2654.956112] LustreError: 107513:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2654.982487] LustreError: 107513:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2654.990329] LustreError: 103618:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2655.139103] LustreError: 107526:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.154@tcp: cfg command failed: rc = -17 [ 2655.150584] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2655.150584] [ 2658.036735] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 2659.334747] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2668.197581] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 05:43:48 (1787564628) [ 2686.400573] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 2703.637034] Lustre: 109557:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2725.721194] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2729.105579] Lustre: 110692:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2807.265684] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2807.274795] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2807.284291] Lustre: Skipped 2 previous similar messages [ 2807.288926] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2807.303566] Lustre: Skipped 5 previous similar messages [ 2809.022921] Lustre: server umount lustre-MDT0000 complete [ 2812.265136] LustreError: 105941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787564775 with bad export cookie 3344887662662328193 [ 2812.266428] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2812.278952] LustreError: 105941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2812.782608] Lustre: server umount lustre-MDT0001 complete [ 2827.022686] Lustre: server umount lustre-OST0000 complete [ 2829.280408] Lustre: 93241:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787564775/real 1787564775] req@ffff9b56f7ad0380 x1874396997567872/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787564791 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2829.314147] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2829.352025] Lustre: Skipped 3 previous similar messages [ 2830.699437] Lustre: server umount lustre-OST0001 complete [ 2835.810443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2844.714225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2852.193388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2861.201645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2879.246993] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 2888.112377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2888.133423] Lustre: lustre-MDT0000: reset Object Index mappings [ 2888.493656] LustreError: 114641:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2888.515618] LustreError: 114641:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2888.603911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2888.610315] Lustre: Skipped 1 previous similar message [ 2893.091429] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 2901.219504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2901.249383] Lustre: lustre-MDT0001: reset Object Index mappings [ 2906.263156] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 2909.034525] Lustre: 115788:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2914.740539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2921.084712] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 2926.787298] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 2929.655976] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2933.331623] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 2933.335784] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 2933.450470] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 2936.804962] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 2943.725125] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2946.954297] Lustre: 117666:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2987.466645] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 05:49:07 (1787564947) [ 3015.157078] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3015.166583] Lustre: Skipped 2 previous similar messages [ 3015.167284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3015.173034] Lustre: Skipped 4 previous similar messages [ 3020.780314] Lustre: server umount lustre-MDT0000 complete [ 3023.659928] LustreError: 114622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787564986 with bad export cookie 3344887662662375065 [ 3023.664104] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3023.673517] LustreError: 114622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3025.378489] LustreError: 114638:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3025.392653] LustreError: 114638:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3030.206552] Lustre: server umount lustre-MDT0001 complete [ 3040.289143] Lustre: server umount lustre-OST0000 complete [ 3049.959292] Lustre: server umount lustre-OST0001 complete [ 3056.483690] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_hostid [ 3065.956208] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 3098.981876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3099.196956] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3099.228648] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3099.337445] Lustre: lustre-MDT0000: new disk, initializing [ 3099.439492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3099.448708] Lustre: Skipped 3 previous similar messages [ 3099.464893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3104.222106] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 3111.135986] Lustre: server umount lustre-MDT0000 complete [ 3120.747495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3121.064408] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3121.079675] Lustre: 121933:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3121.092703] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3125.883888] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 3130.624030] Lustre: server umount lustre-MDT0000 complete [ 3139.307197] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 05:51:39 (1787565099) [ 3140.645988] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3142.711491] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 05:51:42 (1787565102) [ 3157.700932] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 3167.125225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3171.629531] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 3179.236923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3179.343762] Lustre: 123984:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3179.404134] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3179.407781] Lustre: Skipped 1 previous similar message [ 3179.457608] Lustre: lustre-MDT0001: new disk, initializing [ 3179.559295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3179.573338] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3183.238825] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 3187.294703] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3192.690502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3192.942327] Lustre: lustre-OST0000: new disk, initializing [ 3192.948247] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3192.962052] Lustre: 125561:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3194.984092] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3194.993729] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3195.080794] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3198.945358] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 3208.469889] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3208.607731] Lustre: lustre-OST0001: new disk, initializing [ 3208.610728] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3208.615655] Lustre: 126429:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3209.915594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3209.927269] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3210.004974] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3214.087917] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 3223.003702] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3226.452847] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3601.532274] Lustre: 132843:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3601.536363] Lustre: 132843:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3601.545401] Lustre: 132843:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3601.550518] Lustre: 132843:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3601.553654] Lustre: 132843:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3601.557506] Lustre: 132843:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3839.265144] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 06:03:18 (1787565798) [ 3855.329533] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3855.347383] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3855.351557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3855.375748] Lustre: Skipped 5 previous similar messages [ 3855.382422] Lustre: Skipped 9 previous similar messages [ 3860.449520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3860.462084] Lustre: Skipped 4 previous similar messages [ 3860.920976] Lustre: server umount lustre-MDT0000 complete [ 3864.241600] LustreError: 124722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787565826 with bad export cookie 3344887662662440186 [ 3864.247899] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3864.257269] LustreError: 124722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3865.571849] LustreError: 123992:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3865.574222] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3865.587838] LustreError: 123992:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3865.610358] Lustre: Skipped 2 previous similar messages [ 3869.607444] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3869.616604] Lustre: Skipped 3 previous similar messages [ 3870.853266] Lustre: server umount lustre-MDT0001 complete [ 3880.624909] Lustre: server umount lustre-OST0000 complete [ 3890.422871] Lustre: server umount lustre-OST0001 complete [ 3896.248444] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_hostid [ 3905.743706] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 3936.825879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3937.018837] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3937.043378] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3937.117947] Lustre: lustre-MDT0000: new disk, initializing [ 3937.167585] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3937.173598] Lustre: Skipped 5 previous similar messages [ 3937.189480] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3941.496267] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 3951.433189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3951.540533] Lustre: 135895:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3951.585725] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3951.588364] Lustre: Skipped 1 previous similar message [ 3951.649847] Lustre: lustre-MDT0001: new disk, initializing [ 3951.763057] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3951.772430] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3955.504239] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 3965.036946] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3970.337182] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3976.920985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3977.172926] Lustre: lustre-OST0000: new disk, initializing [ 3977.177468] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3977.183619] Lustre: 137863:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3979.122974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3979.154436] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3979.278533] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3983.521813] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 3992.472287] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4000.625379] Lustre: lustre-MDD0000: changelog on [ 4002.733824] Lustre: lustre-MDD0001: changelog on [ 5649.528325] Lustre: lustre-MDD0001: changelog off [ 5651.297551] Lustre: 138256:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 5652.157699] Lustre: lustre-MDD0000: changelog off [ 5658.086456] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5658.096759] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5658.111379] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5658.595864] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5661.665832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5661.676457] Lustre: Skipped 2 previous similar messages [ 5664.128434] Lustre: server umount lustre-OST0000 complete [ 5671.906065] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5671.906876] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5671.919804] Lustre: Skipped 1 previous similar message [ 5671.920579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5671.936856] Lustre: Skipped 1 previous similar message [ 5674.184037] Lustre: server umount lustre-MDT0000 complete [ 5677.026371] LustreError: 135901:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5677.045756] LustreError: 135901:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5677.514585] LustreError: 135888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787567640 with bad export cookie 3344887662663219636 [ 5677.516323] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5677.522894] LustreError: 135888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5677.844753] Lustre: server umount lustre-MDT0001 complete [ 5689.813610] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing unload_modules_local [ 5692.925355] Key type lgssc unregistered [ 5693.281617] LNet: 146960:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5693.294468] LNetError: 146960:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5693.306641] LNet: Removed LNI 192.168.206.154@tcp [ 5694.150233] Key type .llcrypt unregistered [ 5694.153235] Key type ._llcrypt unregistered [ 5713.710231] Key type ._llcrypt registered [ 5713.719197] Key type .llcrypt registered [ 5713.876812] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_hostid [ 5730.958956] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 5731.826347] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5731.906720] alg: No test for adler32 (adler32-zlib) [ 5732.922381] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 5733.103199] LNet: Added LNI 192.168.206.154@tcp [8/256/0/180] [ 5734.768648] Key type lgssc registered [ 5736.237837] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5769.946722] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5769.985139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5771.296495] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5771.322484] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5771.389428] Lustre: lustre-MDT0000: new disk, initializing [ 5771.465691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5771.497878] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5775.625560] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 5784.982745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5785.183845] Lustre: 150167:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5785.271193] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5785.278336] Lustre: Skipped 1 previous similar message [ 5785.382190] Lustre: lustre-MDT0001: new disk, initializing [ 5785.558701] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5785.591503] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5785.598636] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5791.311684] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 5803.317392] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5810.644509] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5817.510130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5817.907626] Lustre: lustre-OST0000: new disk, initializing [ 5817.911783] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5817.921580] Lustre: 152132:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5818.050378] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5824.131174] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5824.149802] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5824.347834] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5825.136344] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 5834.227853] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5840.578108] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_module obdecho/obdecho [ 5842.490686] LustreError: 153137:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 5842.506515] LustreError: 153137:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 5844.505396] LustreError: 153233:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 5857.260689] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5857.275491] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5857.290552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5860.325233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5860.326199] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5862.786377] Lustre: server umount lustre-MDT0000 complete [ 5866.478132] LustreError: 150158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787567829 with bad export cookie 7739159199761515191 [ 5866.483787] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5866.502308] LustreError: 150158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5867.079495] Lustre: server umount lustre-MDT0001 complete [ 5880.762646] Lustre: server umount lustre-OST0000 complete [ 5904.128993] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_hostid [ 5917.448450] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 5956.125776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5956.367639] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5956.395681] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5956.481690] Lustre: lustre-MDT0000: new disk, initializing [ 5956.577052] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5956.603048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5961.512321] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 5971.884943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5972.054759] Lustre: 156950:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5972.133112] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5972.136556] Lustre: Skipped 1 previous similar message [ 5972.199706] Lustre: lustre-MDT0001: new disk, initializing [ 5972.271944] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5972.317172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5972.332518] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5977.076617] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 5985.655287] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5990.172877] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5997.419603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5997.758764] Lustre: lustre-OST0000: new disk, initializing [ 5997.762749] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5997.771987] Lustre: 158919:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5997.852032] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6004.250242] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 6008.026408] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6008.083639] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6008.221539] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6015.387703] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6021.055654] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6021.307222] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6026.559671] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6026.859643] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6033.899706] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6033.901431] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6033.923207] Lustre: Skipped 1 previous similar message [ 6033.928844] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6033.929117] Lustre: Skipped 2 previous similar messages [ 6033.953369] LustreError: Skipped 1 previous similar message [ 6035.083890] Lustre: server umount lustre-OST0000 complete [ 6039.012115] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6039.013235] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6039.024719] Lustre: Skipped 1 previous similar message [ 6039.025690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6039.025695] Lustre: Skipped 1 previous similar message [ 6044.130686] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6044.142553] Lustre: Skipped 2 previous similar messages [ 6044.996279] Lustre: server umount lustre-MDT0000 complete [ 6047.898121] LustreError: 157747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787568010 with bad export cookie 7739159199761516150 [ 6047.903153] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6047.912170] LustreError: 157747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6048.247301] Lustre: server umount lustre-MDT0001 complete [ 6052.296931] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 06:40:11 (1787568011) [ 6053.997843] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6056.081553] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 06:40:15 (1787568015) [ 6063.737475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6064.131502] LustreError: 161332:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6064.147519] LustreError: 161332:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6064.230402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6067.838400] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 6069.221230] LustreError: 161333:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6073.321394] LustreError: 161332:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6076.407524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6082.815303] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 6092.254613] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6097.735185] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6104.843614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6105.106512] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6105.110813] Lustre: Skipped 1 previous similar message [ 6110.528427] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 6116.835432] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6128.915255] Lustre: server umount lustre-MDT0000 complete [ 6131.687069] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6131.693501] LustreError: 161332:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6131.710781] Lustre: Skipped 2 previous similar messages [ 6131.746574] LustreError: 161332:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6133.361480] LustreError: 161314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787568096 with bad export cookie 7739159199761517032 [ 6133.366178] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6133.377040] LustreError: 161314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6133.984513] Lustre: server umount lustre-MDT0001 complete [ 6141.677402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6153.189646] Lustre: 148388:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787568099/real 1787568099] req@0000000048fe8587 x1874400583468800/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787568115 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6153.241489] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6153.260349] Lustre: Skipped 1 previous similar message [ 6154.274505] Lustre: 148388:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787568101/real 1787568101] req@000000004aeda7ea x1874400583468928/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787568117 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6154.994052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6155.145609] LustreError: 165172:0:(llog.c:1655:llog_backup()) MGC192.168.206.154@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6155.152573] Lustre: 165172:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.154@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6157.284084] Lustre: 148390:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787568104/real 1787568104] req@000000004c015026 x1874400583469056/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787568120 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6159.392200] Lustre: 148390:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787568106/real 1787568106] req@00000000324de274 x1874400583469312/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787568122 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6163.566919] LustreError: 165177:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6163.593957] LustreError: 165177:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6164.448172] Lustre: 148387:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787568111/real 1787568111] req@00000000069458fc x1874400583469440/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787568127 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6172.133216] LustreError: 165178:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6172.158372] LustreError: 165178:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6188.513368] LustreError: 165178:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6188.543185] LustreError: 165178:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 6216.162331] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6216.167987] Lustre: 165172:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.154@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6223.841150] LustreError: 165177:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6223.883081] LustreError: 165177:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 24 previous similar messages [ 6256.693851] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6258.733252] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6261.133646] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 6270.182459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6279.253320] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.54@tcp (not set up) [ 6284.045185] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 6284.776646] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6291.078214] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6292.644340] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6298.899486] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6300.278300] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6320.610115] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6320.618973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6320.628668] Lustre: Skipped 2 previous similar messages [ 6320.635054] Lustre: Skipped 1 previous similar message [ 6320.655829] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6326.138827] Lustre: server umount lustre-MDT0000 complete [ 6330.736674] LustreError: 161314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787568293 with bad export cookie 7739159199761523486 [ 6330.749309] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6330.757753] LustreError: 161314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6330.851959] LustreError: 165923:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6330.861820] LustreError: 165923:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 41 previous similar messages [ 6331.241478] Lustre: server umount lustre-MDT0001 complete [ 6337.615349] Lustre: server umount lustre-OST0000 complete [ 6362.835769] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_hostid [ 6374.585521] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 6413.061065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6413.456572] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6413.496850] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6413.612748] Lustre: lustre-MDT0000: new disk, initializing [ 6413.672872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6413.677513] Lustre: Skipped 1 previous similar message [ 6413.692858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6418.509345] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 6428.540165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6428.683308] Lustre: 170281:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6429.060552] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6433.258096] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 6442.482563] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6449.394843] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6455.126671] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6455.368210] Lustre: lustre-OST0000: new disk, initializing [ 6455.370123] Lustre: Skipped 1 previous similar message [ 6455.372635] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6455.375091] Lustre: Skipped 2 previous similar messages [ 6455.380067] Lustre: 172253:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6461.300766] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 6463.528675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6463.536844] Lustre: Skipped 1 previous similar message [ 6463.545333] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6463.623494] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6469.653584] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6474.103811] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6474.294486] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6478.392224] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6478.676274] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6484.449760] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6484.453224] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6484.455839] LustreError: Skipped 1 previous similar message [ 6484.465326] Lustre: Skipped 1 previous similar message [ 6484.472801] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6484.489517] Lustre: Skipped 6 previous similar messages [ 6495.200132] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6495.459270] Lustre: server umount lustre-OST0000 complete [ 6514.144643] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6514.263487] Lustre: server umount lustre-MDT0000 complete [ 6515.200196] LustreError: 170290:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6515.243024] LustreError: 170290:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6518.423256] LustreError: 170275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787568481 with bad export cookie 7739159199761540321 [ 6518.430763] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6518.437142] LustreError: 170275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6518.785450] Lustre: server umount lustre-MDT0001 complete [ 6528.546218] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 06:48:08 (1787568488) [ 6535.226622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6541.064755] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 6549.937106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6555.729644] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 6563.895572] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6570.594539] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6577.649216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6584.748442] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 6593.128290] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6605.067226] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 06:49:25 (1787568565) [ 6610.923466] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6610.920926] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6610.966770] Lustre: Skipped 2 previous similar messages [ 6610.978285] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6610.986241] Lustre: Skipped 10 previous similar messages [ 6617.445890] Lustre: server umount lustre-OST0000 complete [ 6628.916168] Lustre: server umount lustre-MDT0000 complete [ 6632.510616] LustreError: 174646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787568595 with bad export cookie 7739159199761541203 [ 6632.530184] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6633.040670] Lustre: server umount lustre-MDT0001 complete [ 6644.946518] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing unload_modules_local [ 6649.168432] Key type lgssc unregistered [ 6649.488922] LNet: 178396:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6649.503757] LNetError: 178396:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6650.542370] LNet: Removed LNI 192.168.206.154@tcp [ 6651.933799] Key type .llcrypt unregistered [ 6651.935934] Key type ._llcrypt unregistered [ 6671.015867] Key type ._llcrypt registered [ 6671.019191] Key type .llcrypt registered [ 6671.153368] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 6672.360622] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6672.432846] alg: No test for adler32 (adler32-zlib) [ 6673.656512] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 6674.035870] LNet: Added LNI 192.168.206.154@tcp [8/256/0/180] [ 6675.853519] Key type lgssc registered [ 6676.995171] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6688.758568] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6688.773270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6704.290489] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6709.412354] LustreError: 179642:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.154@tcp: failed processing log, type 4: rc = -110 [ 6714.528592] LustreError: 179642:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6714.563384] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 6714.571475] LustreError: MGC192.168.206.154@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 6714.588393] LustreError: 179642:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 6714.619975] LustreError: 179642:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -22 [ 6714.631512] LustreError: 179642:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 6714.698401] Lustre: server umount lustre-OST0000 complete [ 6714.701179] LustreError: 179642:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6723.841483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6725.517050] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6730.699178] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 6735.851048] LustreError: 180142:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6735.888335] LustreError: 180142:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6740.045554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6740.483352] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6745.104986] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 6752.411185] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6759.052187] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6767.626656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6768.074204] LustreError: 181961:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6768.103872] Lustre: 181961:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.206.154@tcp: invalid local config log lustre-OST0000: rc = -22 [ 6768.376563] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6776.492887] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 6783.954589] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6786.529626] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6786.542736] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6786.562684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6791.144964] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6791.174464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6791.176528] Lustre: Skipped 1 previous similar message [ 6791.203918] Lustre: Skipped 2 previous similar messages [ 6792.481263] Lustre: server umount lustre-MDT0000 complete [ 6795.783190] LustreError: 180122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787568758 with bad export cookie 17179630962494125561 [ 6795.800902] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6795.804519] LustreError: 180122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6795.983265] Lustre: server umount lustre-MDT0001 complete [ 6811.400121] Lustre: server umount lustre-OST0000 complete [ 6822.361757] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_hostid [ 6833.747486] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 6869.340938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6869.588333] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6869.617483] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6869.684408] Lustre: lustre-MDT0000: new disk, initializing [ 6869.748394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6869.764721] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6873.990139] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 6883.718699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6883.867982] Lustre: 185484:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6883.921521] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6883.925329] Lustre: Skipped 1 previous similar message [ 6883.987430] Lustre: lustre-MDT0001: new disk, initializing [ 6884.072842] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6884.111592] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6884.123863] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6888.739548] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 6898.894601] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6904.515449] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6911.573303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6912.027532] Lustre: lustre-OST0000: new disk, initializing [ 6912.037579] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6912.053287] Lustre: 187450:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6912.214889] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6918.372738] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 6921.268265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6921.290107] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6921.433159] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6927.426322] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6931.488842] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6931.739779] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6937.121731] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6937.408374] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6942.177983] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6942.184365] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6942.190972] LustreError: Skipped 1 previous similar message [ 6942.192117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6942.196250] Lustre: Skipped 1 previous similar message [ 6945.403715] Lustre: server umount lustre-OST0000 complete [ 6950.883309] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6950.895970] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6950.920661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6950.928545] Lustre: Skipped 1 previous similar message [ 6956.107567] Lustre: server umount lustre-MDT0000 complete [ 6957.537840] LustreError: 185495:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6957.545993] LustreError: 185495:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6959.754923] LustreError: 185474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787568922 with bad export cookie 17179630962494126289 [ 6959.760368] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6959.763264] LustreError: 185474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6960.177798] Lustre: server umount lustre-MDT0001 complete [ 6969.007781] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 06:55:29 (1787568929) [ 6987.525270] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing unload_modules_local [ 6990.573858] Key type lgssc unregistered [ 6990.945503] LNet: 190402:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6990.957721] LNetError: 190402:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6990.970771] LNet: Removed LNI 192.168.206.154@tcp [ 6991.957458] Key type .llcrypt unregistered [ 6991.961815] Key type ._llcrypt unregistered [ 7010.536653] Key type ._llcrypt registered [ 7010.539406] Key type .llcrypt registered [ 7010.698553] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 7012.058104] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7012.130196] alg: No test for adler32 (adler32-zlib) [ 7013.303388] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 7013.504953] LNet: Added LNI 192.168.206.154@tcp [8/256/0/180] [ 7015.212507] Key type lgssc registered [ 7016.651092] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7024.299767] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7024.324255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7025.941127] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7030.673309] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 7035.428140] LustreError: 191619:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7035.446317] LustreError: 191619:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7039.810967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7040.259962] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7045.160437] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 7053.298532] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7059.740881] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7067.788613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7068.091290] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7068.100040] LustreError: 193486:0:(llog.c:1696:llog_backup()) MGC192.168.206.154@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7068.122289] Lustre: 193486:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.206.154@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7068.293435] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7075.582772] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 7083.363892] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7086.566653] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7086.579958] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7086.599778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7090.146025] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7090.150302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7090.174758] Lustre: Skipped 2 previous similar messages [ 7092.203823] Lustre: server umount lustre-MDT0000 complete [ 7095.268856] LustreError: 192348:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7095.290129] LustreError: 192348:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7096.544812] LustreError: 191602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787569059 with bad export cookie 12216000184035759715 [ 7096.562049] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7096.563396] LustreError: 191602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7097.038170] Lustre: server umount lustre-MDT0001 complete [ 7112.184721] Lustre: server umount lustre-OST0000 complete [ 7122.895289] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_hostid [ 7134.501656] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 7172.585182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7172.806307] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7172.849868] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7172.918893] Lustre: lustre-MDT0000: new disk, initializing [ 7172.975083] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7172.991702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7177.589282] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 7187.450634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7187.620913] Lustre: 197007:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7187.662373] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7187.666307] Lustre: Skipped 1 previous similar message [ 7187.670252] LustreError: 197790:0:(llog.c:1696:llog_backup()) MGC192.168.206.154@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7187.685820] LustreError: 197790:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7187.702084] Lustre: 197790:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.154@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7187.736120] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7187.746152] Lustre: Skipped 1 previous similar message [ 7187.857985] Lustre: lustre-MDT0001: new disk, initializing [ 7187.984916] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7188.025784] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7188.056235] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7192.715978] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 7202.297987] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7207.776674] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7215.222792] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7215.539691] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7215.544149] Lustre: Skipped 1 previous similar message [ 7215.553245] LustreError: 198952:0:(llog.c:1696:llog_backup()) MGC192.168.206.154@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7215.562601] LustreError: 198952:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7215.568155] Lustre: 198952:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.154@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7215.575139] Lustre: 198952:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7215.593683] Lustre: lustre-OST0000: new disk, initializing [ 7215.597908] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7215.604405] Lustre: 198976:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7215.702919] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7223.089387] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 7223.883918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7223.900860] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7224.064103] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7232.414665] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7236.578671] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7236.728662] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7240.924264] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7241.224891] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7244.770623] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7244.772808] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7244.782835] LustreError: Skipped 1 previous similar message [ 7244.785964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7244.813766] Lustre: Skipped 2 previous similar messages [ 7244.849631] Lustre: Skipped 1 previous similar message [ 7249.963607] Lustre: server umount lustre-OST0000 complete [ 7254.499093] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7254.519017] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7254.544857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7260.130947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7260.148017] Lustre: Skipped 3 previous similar messages [ 7260.888910] Lustre: server umount lustre-MDT0000 complete [ 7264.120971] LustreError: 196999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787569226 with bad export cookie 12216000184035760436 [ 7264.128062] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7264.133102] LustreError: 196999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7264.401279] Lustre: server umount lustre-MDT0001 complete [ 7274.799160] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 07:00:34 (1787569234) [ 7291.324596] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 7301.327035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7301.691660] LustreError: 202114:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7301.732923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7305.663824] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 7307.237940] LustreError: 202115:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7311.342360] LustreError: 202114:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7313.863199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7319.025188] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 7321.910829] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7328.211897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7334.865687] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 7344.196106] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7344.404479] Lustre: lustre-OST0001: new disk, initializing [ 7344.409149] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7344.415239] Lustre: 204421:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7344.499943] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7344.510658] Lustre: Skipped 2 previous similar messages [ 7350.786808] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 7354.451685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7354.459977] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7354.588690] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7361.324290] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7371.795381] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7373.750346] Lustre: DEBUG MARKER: ADD OST3 [ 7378.963818] loop: module loaded [ 7379.020853] loop0: detected capacity change from 0 to 204800000 [ 7379.252143] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7379.269189] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7379.291824] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7385.276181] loop0: detected capacity change from 0 to 204800000 [ 7386.391856] Lustre: DEBUG MARKER: STOP OST3 [ 7391.692321] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7391.937091] Lustre: lustre-OST0003: new disk, initializing [ 7391.943245] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7391.955234] Lustre: 206418:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7393.683574] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7393.835455] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7393.852747] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7393.889928] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7395.744119] LustreError: 206591:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7396.155598] Lustre: Failing over lustre-OST0003 [ 7396.375360] Lustre: server umount lustre-OST0003 complete [ 7398.916418] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7398.921555] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7398.926250] Lustre: Skipped 1 previous similar message [ 7398.931547] LustreError: 203600:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7398.963254] LustreError: 203600:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7399.074934] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 7407.732512] Lustre: DEBUG MARKER: START OST3 again [ 7414.443964] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7414.757475] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7414.778576] Lustre: Skipped 1 previous similar message [ 7414.793346] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7416.487257] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7416.675910] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 7416.676342] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7421.000767] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 7441.966653] Lustre: Failing over lustre-OST0003 [ 7442.113708] Lustre: server umount lustre-OST0003 complete [ 7442.408210] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7442.411870] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7442.422134] LustreError: Skipped 1 previous similar message [ 7442.423279] LustreError: 203600:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7442.423288] LustreError: 203600:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 7442.503718] Lustre: Skipped 2 previous similar messages [ 7460.839127] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7460.841426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7460.856017] Lustre: Skipped 2 previous similar messages [ 7460.861414] Lustre: Skipped 1 previous similar message [ 7474.144991] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7474.335035] Lustre: server umount lustre-MDT0000 complete [ 7476.196467] LustreError: 202114:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7476.218739] LustreError: 202114:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 7477.510547] LustreError: 206414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787569440 with bad export cookie 12216000184035761318 [ 7477.532779] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7477.535088] LustreError: 206414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 7481.321401] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7481.330239] Lustre: Skipped 10 previous similar messages [ 7484.105429] Lustre: server umount lustre-MDT0001 complete [ 7487.751463] Lustre: server umount lustre-OST0000 complete [ 7491.113421] Lustre: server umount lustre-OST0001 complete [ 7496.824850] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_hostid [ 7504.203709] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 7538.516615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7538.710087] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7538.742089] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7538.807440] Lustre: lustre-MDT0000: new disk, initializing [ 7538.880811] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7544.923359] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 7557.693186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7557.889566] Lustre: 211881:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7557.946859] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7557.949341] Lustre: Skipped 1 previous similar message [ 7557.998590] Lustre: lustre-MDT0001: new disk, initializing [ 7558.097141] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7558.101062] Lustre: Skipped 1 previous similar message [ 7558.131047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7558.149372] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7564.402143] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 7574.186685] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7580.562582] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7588.182984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7588.562260] Lustre: 213851:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7590.272377] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7590.294853] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7590.474128] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7596.594735] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 7607.083738] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7612.448476] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7612.648819] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7617.876229] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7618.148054] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7621.094522] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7621.096460] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7621.110271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7621.110280] Lustre: Skipped 3 previous similar messages [ 7621.133029] Lustre: Skipped 4 previous similar messages [ 7626.713536] Lustre: server umount lustre-OST0000 complete [ 7645.152285] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7645.418552] Lustre: server umount lustre-MDT0000 complete [ 7646.694677] LustreError: 211888:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7646.710021] LustreError: 211888:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 7649.215310] LustreError: 211872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787569611 with bad export cookie 12216000184035763614 [ 7649.219135] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7649.227704] LustreError: 211872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7649.760844] Lustre: server umount lustre-MDT0001 complete [ 7654.061882] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 07:06:53 (1787569613) [ 7671.265694] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_hostid [ 7683.259730] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 7718.771127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7719.003822] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7719.029920] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7719.032925] Lustre: Skipped 1 previous similar message [ 7719.105853] Lustre: lustre-MDT0000: new disk, initializing [ 7719.108752] Lustre: Skipped 1 previous similar message [ 7719.199914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7724.034984] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 7734.511062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7734.616779] Lustre: 218465:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7734.910031] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7739.715669] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 7748.486202] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7753.989660] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7761.587787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7762.003083] Lustre: 220434:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7763.709286] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7769.537888] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 7780.311755] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7785.461806] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7785.778992] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7791.563751] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7791.817754] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7794.680756] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7794.681622] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7794.682619] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7794.682625] Lustre: Skipped 9 previous similar messages [ 7794.685873] LustreError: Skipped 1 previous similar message [ 7794.741066] Lustre: Skipped 3 previous similar messages [ 7799.801979] Lustre: server umount lustre-OST0000 complete [ 7808.754324] Lustre: server umount lustre-MDT0000 complete [ 7810.017480] LustreError: 218470:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7810.032577] LustreError: 218470:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7812.197294] LustreError: 218455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787569774 with bad export cookie 12216000184035764496 [ 7812.210700] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7812.218737] LustreError: 218455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7812.522466] Lustre: server umount lustre-MDT0001 complete [ 7821.189702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7821.723994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7821.728231] Lustre: Skipped 4 previous similar messages [ 7826.153270] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 7834.051776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7838.934761] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 7846.095294] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7851.433901] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7857.386625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7864.230216] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 7871.823904] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7882.743156] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7887.735830] Lustre: server umount lustre-OST0000 complete [ 7898.092403] Lustre: server umount lustre-MDT0000 complete [ 7901.418439] LustreError: 222633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787569864 with bad export cookie 12216000184035765378 [ 7901.424149] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7901.433910] LustreError: 222633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7901.815523] Lustre: server umount lustre-MDT0001 complete [ 8013.223120] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing unload_modules_local [ 8017.190695] Key type lgssc unregistered [ 8017.545696] LNet: 226317:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8017.564645] LNetError: 226317:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8017.599075] LNet: Removed LNI 192.168.206.154@tcp [ 8018.533158] Key type .llcrypt unregistered [ 8018.535156] Key type ._llcrypt unregistered [ 8042.140555] Key type ._llcrypt registered [ 8042.147047] Key type .llcrypt registered [ 8042.257353] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_hostid [ 8057.559223] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 8058.912681] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8059.037874] alg: No test for adler32 (adler32-zlib) [ 8060.201354] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 8060.445811] LNet: Added LNI 192.168.206.154@tcp [8/256/0/180] [ 8062.112200] Key type lgssc registered [ 8063.163554] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8096.352382] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8096.402506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8097.662906] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8097.699336] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8097.801959] Lustre: lustre-MDT0000: new disk, initializing [ 8097.942281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8097.973823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8102.608690] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 8112.604815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8112.732825] Lustre: 229669:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8112.807883] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8112.817650] Lustre: Skipped 1 previous similar message [ 8112.965754] Lustre: lustre-MDT0001: new disk, initializing [ 8113.132641] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8113.173222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8113.191521] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8118.637518] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 8128.566237] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8134.359393] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8141.254194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8141.540067] Lustre: lustre-OST0000: new disk, initializing [ 8141.546747] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8141.552858] Lustre: 231639:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8141.657848] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8148.196298] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 8148.618031] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8148.646647] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8148.849344] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8156.969515] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8161.186884] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8161.354522] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8165.266298] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8165.415880] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8169.473639] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8169.483332] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8169.490079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8169.497250] Lustre: Skipped 1 previous similar message [ 8169.527469] LustreError: Skipped 1 previous similar message [ 8173.904319] Lustre: server umount lustre-OST0000 complete [ 8179.681385] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8179.690031] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8179.692666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8179.692673] Lustre: Skipped 1 previous similar message [ 8179.717641] Lustre: Skipped 1 previous similar message [ 8184.637268] Lustre: server umount lustre-MDT0000 complete [ 8188.310123] LustreError: 229663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787570151 with bad export cookie 15526172431615246411 [ 8188.321434] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8188.326654] LustreError: 229663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8188.640454] Lustre: server umount lustre-MDT0001 complete [ 8209.036731] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 8219.204272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8219.647299] LustreError: 234572:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8219.671970] LustreError: 234572:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8219.705892] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8223.533486] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 8224.739958] LustreError: 234573:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8229.859964] LustreError: 234572:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8231.430569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8231.704420] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8235.907654] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 8238.141626] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8243.789053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8244.191909] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8251.586078] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 8260.334780] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8260.548512] Lustre: lustre-OST0001: new disk, initializing [ 8260.552035] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8260.560423] Lustre: 236879:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8260.670400] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8265.741680] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 8269.366528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8269.375723] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8269.444657] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8276.432890] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8284.214831] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8298.219639] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 07:17:38 (1787570258) [ 8303.019849] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8303.025401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8303.042235] Lustre: Skipped 2 previous similar messages [ 8303.054010] Lustre: Skipped 4 previous similar messages [ 8306.599163] Lustre: server umount lustre-MDT0000 complete [ 8308.194861] LustreError: 234573:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8308.237898] LustreError: 234573:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8310.494436] LustreError: 237648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787570273 with bad export cookie 15526172431615247293 [ 8310.502528] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8310.509481] LustreError: 237648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8310.838867] Lustre: server umount lustre-MDT0001 complete [ 8324.406107] Lustre: server umount lustre-OST0000 complete [ 8338.233446] Lustre: server umount lustre-OST0001 complete [ 8345.355593] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_hostid [ 8356.533085] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 8392.131722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8392.512619] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8392.550218] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8392.673329] Lustre: lustre-MDT0000: new disk, initializing [ 8392.769400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8392.802618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8397.898640] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 8408.811299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8408.994593] Lustre: 240824:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8409.116301] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8409.120363] Lustre: Skipped 1 previous similar message [ 8409.179651] Lustre: lustre-MDT0001: new disk, initializing [ 8409.346431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8409.356001] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8413.288258] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 8422.391511] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8427.553446] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8433.515726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8433.800612] Lustre: lustre-OST0000: new disk, initializing [ 8433.809371] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8433.819983] Lustre: 242794:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8435.572261] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8435.598418] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8435.776839] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8440.269518] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 8449.425855] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8453.443528] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8453.697601] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8458.100504] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8458.275642] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8461.283244] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8461.283746] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8461.289063] LustreError: Skipped 1 previous similar message [ 8461.290761] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8461.297789] Lustre: Skipped 2 previous similar messages [ 8461.306105] Lustre: Skipped 1 previous similar message [ 8466.183127] Lustre: server umount lustre-OST0000 complete [ 8471.011446] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8471.018153] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8471.025955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8481.784253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8481.795790] Lustre: Skipped 4 previous similar messages [ 8483.296456] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8483.483799] Lustre: server umount lustre-MDT0000 complete [ 8486.335773] LustreError: 240817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787570449 with bad export cookie 15526172431615249428 [ 8486.338993] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8486.343057] LustreError: 240817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8486.625766] Lustre: server umount lustre-MDT0001 complete [ 8494.536711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8495.171435] LustreError: 245011:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8495.265889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8495.279768] Lustre: Skipped 2 previous similar messages [ 8499.897791] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 8504.810975] LustreError: 245012:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8504.833451] LustreError: 245012:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8508.627751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8513.341216] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 8520.797520] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8526.172689] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8532.857269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8539.161111] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 8545.349808] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8693.221016] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8693.232265] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8693.233159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8693.239618] Lustre: Skipped 2 previous similar messages [ 8693.247495] LustreError: Skipped 1 previous similar message [ 8696.115676] Lustre: server umount lustre-OST0000 complete [ 8703.466114] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8705.785803] Lustre: server umount lustre-MDT0000 complete [ 8708.584888] LustreError: 245012:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8708.608123] LustreError: 245012:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8709.355943] LustreError: 244991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787570672 with bad export cookie 15526172431615250303 [ 8709.374972] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8709.379740] LustreError: 244991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8709.726819] Lustre: server umount lustre-MDT0001 complete [ 8815.098083] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing unload_modules_local [ 8818.263715] Key type lgssc unregistered [ 8818.573434] LNet: 248625:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8818.584194] LNetError: 248625:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8818.611264] LNet: Removed LNI 192.168.206.154@tcp [ 8819.766479] Key type .llcrypt unregistered [ 8819.768709] Key type ._llcrypt unregistered [ 8844.434693] Key type ._llcrypt registered [ 8844.437833] Key type .llcrypt registered [ 8844.560906] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_hostid [ 8860.643969] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 8861.709273] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8861.782078] alg: No test for adler32 (adler32-zlib) [ 8862.954112] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 8863.159527] LNet: Added LNI 192.168.206.154@tcp [8/256/0/180] [ 8864.842335] Key type lgssc registered [ 8866.004370] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8909.010427] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 8917.798061] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8917.829708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8919.076924] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8919.107797] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8919.186609] Lustre: lustre-MDT0000: new disk, initializing [ 8919.249253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8919.264714] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8923.617507] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 8932.877822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8932.998253] Lustre: 252692:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8933.093568] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8933.098558] Lustre: Skipped 1 previous similar message [ 8933.178206] Lustre: lustre-MDT0001: new disk, initializing [ 8933.263974] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8933.313246] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8933.320733] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8937.325649] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 8941.749553] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8947.859285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8948.209842] Lustre: lustre-OST0000: new disk, initializing [ 8948.224386] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8948.230697] Lustre: 254325:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8948.344788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8951.164907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8951.189429] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8951.282879] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8954.705342] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 8965.904086] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8966.190786] Lustre: lustre-OST0001: new disk, initializing [ 8966.198495] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8966.214577] Lustre: 255195:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8966.386962] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8974.114484] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 8974.960154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8974.974665] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8975.153682] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8986.441486] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8992.966982] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9005.025116] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9005.039751] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9005.084175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9006.058950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9006.059747] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9011.174990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9011.186970] Lustre: Skipped 6 previous similar messages [ 9016.290261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9016.299346] Lustre: Skipped 1 previous similar message [ 9019.360740] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9019.644809] Lustre: server umount lustre-MDT0000 complete [ 9023.671452] LustreError: 252685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787570986 with bad export cookie 6679763982381732439 [ 9023.680256] LustreError: 252685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9023.688796] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9024.094912] Lustre: server umount lustre-MDT0001 complete [ 9039.124290] Lustre: server umount lustre-OST0000 complete [ 9052.777471] Lustre: server umount lustre-OST0001 complete [ 9066.512257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9076.944432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9086.240813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9096.403973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9117.006626] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 9127.377892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9127.406326] Lustre: lustre-MDT0000: reset Object Index mappings [ 9128.008535] LustreError: 260210:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9128.036827] LustreError: 260210:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9128.081760] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9133.236331] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 9133.540301] LustreError: 260211:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9137.650238] LustreError: 260210:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9141.909109] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9141.937846] Lustre: lustre-MDT0001: reset Object Index mappings [ 9142.366790] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9147.087617] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 9149.758247] Lustre: 261351:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9156.520679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9162.806269] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 9168.241758] LustreError: 262374:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9170.802926] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9171.088237] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9171.095277] Lustre: Skipped 1 previous similar message [ 9173.187061] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9173.187931] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9177.691577] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 9185.310714] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9188.900761] Lustre: 263224:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9193.443461] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9193.450019] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9193.468365] Lustre: Skipped 2 previous similar messages [ 9193.473989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9193.477626] Lustre: Skipped 2 previous similar messages [ 9195.673713] Lustre: server umount lustre-MDT0000 complete [ 9199.086914] LustreError: 260210:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9199.115885] LustreError: 260210:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9199.141080] LustreError: 260191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787571161 with bad export cookie 6679763982381739222 [ 9199.146406] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9199.149773] LustreError: 260191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9199.550480] Lustre: server umount lustre-MDT0001 complete [ 9213.684152] Lustre: server umount lustre-OST0000 complete [ 9227.915308] Lustre: server umount lustre-OST0001 complete [ 9246.691412] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 07:33:26 (1787571206) [ 9261.250708] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_hostid [ 9272.055746] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 9312.806153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9313.079441] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9313.134454] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9313.222432] Lustre: lustre-MDT0000: new disk, initializing [ 9313.286823] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9313.316406] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9318.154885] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 9328.799966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9328.918608] Lustre: 267478:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9328.997558] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9329.002945] Lustre: Skipped 1 previous similar message [ 9329.086797] Lustre: lustre-MDT0001: new disk, initializing [ 9329.229114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9329.239186] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9333.477037] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 9343.208471] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9348.422675] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9354.714213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9355.027902] Lustre: lustre-OST0000: new disk, initializing [ 9355.032387] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9355.043921] Lustre: 269446:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9356.314402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9356.337286] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9356.504564] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9361.928394] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 9371.192801] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9375.399746] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9375.565586] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9379.079910] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9379.297420] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9381.859674] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9381.863135] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9381.872620] Lustre: Skipped 3 previous similar messages [ 9381.881945] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9381.890924] Lustre: Skipped 3 previous similar messages [ 9395.168231] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9395.347852] Lustre: server umount lustre-OST0000 complete [ 9400.801441] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9400.820151] LustreError: Skipped 1 previous similar message [ 9400.830217] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9400.855555] Lustre: Skipped 1 previous similar message [ 9400.861146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9400.870115] Lustre: Skipped 6 previous similar messages [ 9405.056692] Lustre: server umount lustre-MDT0000 complete [ 9407.971767] LustreError: 267484:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9408.002519] LustreError: 267484:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9408.449986] LustreError: 267470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787571371 with bad export cookie 6679763982381741322 [ 9408.455877] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9408.468596] LustreError: 267470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9408.733260] Lustre: server umount lustre-MDT0001 complete [ 9426.769951] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 9435.471873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9435.767473] LustreError: 272383:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9435.804350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9435.812382] Lustre: Skipped 2 previous similar messages [ 9438.894090] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 9446.460350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9451.018896] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 9453.676648] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9459.771138] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9466.510864] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 9473.984393] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9474.141153] Lustre: lustre-OST0001: new disk, initializing [ 9474.144401] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9474.150624] Lustre: 274690:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9475.835504] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9475.845773] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9475.940471] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9480.336307] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 9488.913358] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9492.412163] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9497.435774] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 9497.527199] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 9502.965965] LustreError: 250202:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9b56e772e300 x1874403865619968/t0(0) o6->lustre-OST0001-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-1-0.0' uid:0 gid:0 projid:4294967295 [ 9502.998637] LustreError: 250202:0:(client.c:1394:ptlrpc_import_delay_req()) Skipped 2 previous similar messages [ 9503.194959] Lustre: server umount lustre-MDT0000 complete [ 9503.208328] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9503.218558] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9503.230135] Lustre: Skipped 1 previous similar message [ 9503.240827] LustreError: 272383:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9503.250735] LustreError: 272383:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9506.142337] LustreError: 272364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787571468 with bad export cookie 6679763982381742204 [ 9506.145428] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9506.160308] LustreError: 272364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9506.789090] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9506.800575] Lustre: Skipped 2 previous similar messages [ 9508.471735] Lustre: server umount lustre-MDT0001 complete [ 9512.089941] Lustre: server umount lustre-OST0000 complete [ 9515.201975] Lustre: server umount lustre-OST0001 complete [ 9528.959905] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 9537.711771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9542.106324] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 9549.371616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9553.773430] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 9556.113847] Lustre: 278882:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9561.846236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9564.277078] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 9567.719224] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 9569.768331] LustreError: 279392:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9569.778216] LustreError: 279392:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 9574.514502] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9574.725502] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9574.731070] Lustre: Skipped 6 previous similar messages [ 9575.742283] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 9579.935301] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 9585.943501] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9588.833131] Lustre: 280758:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9592.462677] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 9592.565465] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 9598.890837] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 07:39:19 (1787571559) [ 9616.353279] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9616.365595] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9616.375124] Lustre: Skipped 4 previous similar messages [ 9616.383107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9626.080467] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9626.271221] Lustre: server umount lustre-MDT0000 complete [ 9629.708194] LustreError: 280067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787571592 with bad export cookie 6679763982381748567 [ 9629.718824] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9629.727593] LustreError: 280067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9636.092921] Lustre: server umount lustre-MDT0001 complete [ 9645.526911] Lustre: server umount lustre-OST0000 complete [ 9655.023982] Lustre: server umount lustre-OST0001 complete [ 9660.572214] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_hostid [ 9668.633494] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 9704.283737] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 9712.612421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9712.822936] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9712.852882] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9712.939850] Lustre: lustre-MDT0000: new disk, initializing [ 9713.009625] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9716.713464] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 9725.712417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9725.838192] Lustre: 284995:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9726.049764] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9729.612810] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 9738.683513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9738.865880] Lustre: 286626:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9740.236608] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9744.071479] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 9752.886059] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9753.020128] Lustre: 287498:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9754.859970] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9757.734603] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 9765.334610] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9768.125746] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9768.130828] Lustre: Skipped 1 previous similar message [ 9781.642691] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 07:42:21 (1787571741) [ 9785.829284] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9785.838184] Lustre: Skipped 6 previous similar messages [ 9785.841579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9785.844847] Lustre: Skipped 11 previous similar messages [ 9789.728800] Lustre: server umount lustre-MDT0000 complete [ 9790.957827] LustreError: 285007:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9790.965860] LustreError: 285007:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 9792.870746] LustreError: 284989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787571755 with bad export cookie 6679763982381756463 [ 9792.880579] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9792.888322] LustreError: 284989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9793.425925] Lustre: server umount lustre-MDT0001 complete [ 9807.295702] Lustre: server umount lustre-OST0000 complete [ 9820.971147] Lustre: server umount lustre-OST0001 complete [ 9828.120926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9832.400475] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 9839.122690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9839.471319] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9839.476293] Lustre: Skipped 5 previous similar messages [ 9842.772706] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 9849.213927] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9855.144365] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9861.044665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9867.232327] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 9873.909633] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9888.517282] loop1: detected capacity change from 0 to 204800000 [ 9888.780243] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9888.792617] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9888.801378] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9893.328507] loop1: detected capacity change from 0 to 204800000 [ 9898.416337] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9898.494137] LustreError: 289730:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs9820: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 9898.502247] LustreError: 289730:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs9820 [ 9898.505571] LustreError: 289730:0:(mgs_handler.c:517:mgs_target_reg()) fs9820-MDT0001: Error checking target index: rc = -13 [ 9898.516942] LustreError: fs9820-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 9898.525307] LustreError: 293460:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [ 9898.530811] LustreError: 293460:0:(tgt_mount.c:2132:server_put_super()) no obd fs9820-MDT0001 [ 9898.534506] LustreError: 293460:0:(tgt_mount.c:116:server_deregister_mount()) fs9820-MDT0001 not registered [ 9898.564690] Lustre: server umount fs9820-MDT0001 complete [ 9898.572401] LustreError: 293460:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 9905.527610] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9905.582428] Lustre: Setting parameter fs9820-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs9820-MDT0001 [ 9905.630707] Lustre: srv-fs9820-MDT0001: No data found on store. Initialize space. [ 9905.635632] Lustre: Skipped 4 previous similar messages [ 9905.691083] Lustre: fs9820-MDT0001: new disk, initializing [ 9905.695799] Lustre: Skipped 3 previous similar messages [ 9905.796084] Lustre: srv-fs9820-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 9905.802310] Lustre: 293928:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs9820-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9905.808756] LustreError: 293928:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs9820-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9909.178805] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 9915.088564] Lustre: server umount fs9820-MDT0001 complete [ 9927.184491] loop1: detected capacity change from 0 to 204800000 [ 9932.388596] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9932.782868] Lustre: 295493:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs9820-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9932.791401] LustreError: 295493:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs9820-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9936.963940] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 9941.031675] Lustre: server umount fs9820-MDT0001 complete [ 9950.690340] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9950.696603] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [ 9950.698804] Lustre: Skipped 3 previous similar messages [ 9954.851731] Lustre: server umount lustre-MDT0000 complete [ 9962.345541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9962.459097] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9966.164855] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 9968.141419] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9973.580883] loop1: detected capacity change from 0 to 204800000 [ 9978.148693] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9978.366275] Lustre: 297940:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs9820-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9978.385505] LustreError: 297940:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs9820-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9980.966184] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 9984.650668] Lustre: server umount fs9820-MDT0001 complete [10002.915417] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10016.225143] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10016.345238] Lustre: server umount lustre-OST0000 complete [10025.703202] Lustre: server umount lustre-MDT0000 complete [10029.459195] LustreError: 289724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787571992 with bad export cookie 6679763982381759144 [10029.464458] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10029.476766] LustreError: 289724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10030.090305] Lustre: server umount lustre-MDT0001 complete [10041.956374] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing unload_modules_local [10044.782825] Key type lgssc unregistered [10045.087738] LNet: 300287:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10045.097542] LNetError: 300287:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10045.118117] LNet: Removed LNI 192.168.206.154@tcp [10045.992159] Key type .llcrypt unregistered [10045.993691] Key type ._llcrypt unregistered [10063.828259] Key type ._llcrypt registered [10063.829883] Key type .llcrypt registered [10063.922104] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [10065.377379] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10065.429497] alg: No test for adler32 (adler32-zlib) [10066.472677] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [10066.813587] LNet: Added LNI 192.168.206.154@tcp [8/256/0/180] [10068.544976] Key type lgssc registered [10069.778148] Lustre: Echo OBD driver; http://www.lustre.org/ [10077.340919] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10077.365290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10078.758181] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10083.253432] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [10088.931450] LustreError: 301559:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10088.957351] LustreError: 301559:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10090.698205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10091.151938] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10094.879882] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [10101.857514] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10107.377274] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10113.155175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10113.517989] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10115.563547] LustreError: 303565:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10119.461276] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [10120.677180] LustreError: 303390:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10123.774774] LustreError: 303390:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10123.801787] LustreError: 303390:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10126.883321] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10128.903285] LustreError: 303389:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10128.953791] LustreError: 303389:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10139.116641] LustreError: 303565:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10139.144223] LustreError: 303565:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10147.642080] loop1: detected capacity change from 0 to 204800000 [10147.829574] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10147.846181] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10147.860942] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10152.999277] loop1: detected capacity change from 0 to 204800000 [10155.611233] LustreError: 303390:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10155.648633] LustreError: 303390:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [10158.553628] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10158.655314] Lustre: fs3937-OST0000: new disk, initializing [10158.659316] Lustre: srv-fs3937-OST0000: No data found on store. Initialize space. [10158.666092] Lustre: 305293:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10158.734725] Lustre: fs3937-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10164.065698] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [10171.237644] Lustre: server umount fs3937-OST0000 complete [10183.989370] loop1: detected capacity change from 0 to 204800000 [10190.286413] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10190.326952] LustreError: 303388:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10190.358467] LustreError: 303388:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 19 previous similar messages [10190.666772] Lustre: fs3937-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10198.446833] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [10200.557889] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10205.705693] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10208.141641] loop2: detected capacity change from 0 to 204800000 [10208.400587] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10208.416114] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10208.450501] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10213.190256] loop2: detected capacity change from 0 to 204800000 [10218.030851] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10218.133154] LustreError: fs3937-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [10218.141710] LustreError: fs3937-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10218.151683] LustreError: 308331:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [10218.157371] LustreError: 308331:0:(tgt_mount.c:2132:server_put_super()) no obd fs3937-OST0001 [10218.162420] LustreError: 308331:0:(tgt_mount.c:116:server_deregister_mount()) fs3937-OST0001 not registered [10218.192495] Lustre: server umount fs3937-OST0001 complete [10218.200396] LustreError: 308331:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10227.650973] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10227.879723] Lustre: fs3937-OST0001: new disk, initializing [10227.883729] Lustre: srv-fs3937-OST0001: No data found on store. Initialize space. [10227.888751] Lustre: 308882:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10227.934150] Lustre: fs3937-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10233.467502] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [10240.930331] Lustre: server umount fs3937-OST0000 complete [10249.267858] Lustre: server umount fs3937-OST0001 complete [10256.865656] LustreError: 303390:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10256.881053] LustreError: 303390:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 37 previous similar messages [10260.347633] loop1: detected capacity change from 0 to 204800000 [10265.587228] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10265.721047] Lustre: fs3937-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10271.689549] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [10278.606728] loop2: detected capacity change from 0 to 204800000 [10283.812555] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10290.501055] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [10299.100842] Lustre: server umount fs3937-OST0000 complete [10309.981824] Lustre: server umount fs3937-OST0001 complete [10325.474652] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10325.487060] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10325.506489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10328.548771] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10328.550864] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10329.906905] Lustre: server umount lustre-OST0000 complete [10333.666922] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10333.668536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10333.685939] Lustre: Skipped 1 previous similar message [10333.692829] Lustre: Skipped 1 previous similar message [10338.786686] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10338.798189] Lustre: Skipped 2 previous similar messages [10339.865838] Lustre: server umount lustre-MDT0000 complete [10343.179933] LustreError: 310131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787572305 with bad export cookie 919822881055111250 [10343.190500] LustreError: 310131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10343.192699] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10343.520219] Lustre: server umount lustre-MDT0001 complete [10353.480189] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing unload_modules_local [10356.264936] Key type lgssc unregistered [10356.584778] LNet: 314811:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10356.594571] LNetError: 314811:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10356.621695] LNet: Removed LNI 192.168.206.154@tcp [10357.784351] Key type .llcrypt unregistered [10357.789202] Key type ._llcrypt unregistered [10388.196621] Key type ._llcrypt registered [10388.198608] Key type .llcrypt registered [10388.286541] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_hostid [10403.571526] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [10404.558746] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10404.574128] alg: No test for adler32 (adler32-zlib) [10405.731096] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [10405.991208] LNet: Added LNI 192.168.206.154@tcp [8/256/0/180] [10407.712316] Key type lgssc registered [10408.657387] Lustre: Echo OBD driver; http://www.lustre.org/ [10442.250199] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10442.273059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10443.557612] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10443.608759] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10443.710448] Lustre: lustre-MDT0000: new disk, initializing [10443.800352] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10443.824900] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10448.351758] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [10458.241454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10458.333291] Lustre: 318669:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10458.432513] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10458.437547] Lustre: Skipped 1 previous similar message [10458.536473] Lustre: lustre-MDT0001: new disk, initializing [10458.650873] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10458.708716] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10458.727919] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10462.825438] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [10472.614902] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10477.944203] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10484.188585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10484.498749] Lustre: lustre-OST0000: new disk, initializing [10484.504704] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10484.511231] Lustre: 320640:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10484.584515] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10490.668240] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [10493.475840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10493.484929] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10493.562336] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10499.154739] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10501.075979] Lustre: Mounted lustre-client - version 2.17.57_83_g6b3cfd8 [10501.919623] Lustre: Unmounted lustre-client [10513.889310] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10513.901103] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10513.928533] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10514.401393] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10514.408121] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10516.474505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10516.481868] Lustre: Skipped 2 previous similar messages [10518.686480] Lustre: server umount lustre-OST0000 complete [10525.155318] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10525.170988] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10525.208940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10528.724938] Lustre: server umount lustre-MDT0000 complete [10532.515786] LustreError: 318661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787572495 with bad export cookie 9530269684924379625 [10532.533438] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10532.534903] LustreError: 318661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10532.785889] Lustre: server umount lustre-MDT0001 complete [10543.738360] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing unload_modules_local [10546.619788] Key type lgssc unregistered [10562.401644] LNet: 322803:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10562.427162] LNetError: 322803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10631.081721] LNet: Removed LNI 192.168.206.154@tcp [10632.166562] Key type .llcrypt unregistered [10632.169894] Key type ._llcrypt unregistered [10650.575803] Key type ._llcrypt registered [10650.584842] Key type .llcrypt registered [10650.704383] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_hostid [10667.647555] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [10668.565302] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10668.619058] alg: No test for adler32 (adler32-zlib) [10670.018354] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [10670.269238] LNet: Added LNI 192.168.206.154@tcp [8/256/0/180] [10672.000161] Key type lgssc registered [10673.520938] Lustre: Echo OBD driver; http://www.lustre.org/ [10705.036818] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 07:57:45 (1787572665) [10711.184593] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10711.228025] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10712.464587] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10712.487752] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10712.548855] Lustre: lustre-MDT0000: new disk, initializing [10712.627465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10712.644472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10717.194667] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [10726.696861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10726.775586] Lustre: 326026:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10726.813603] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10726.817256] Lustre: Skipped 1 previous similar message [10726.872376] Lustre: lustre-MDT0001: new disk, initializing [10726.955254] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10726.998864] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10727.007846] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10730.926291] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [10740.078606] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10745.976496] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10752.727446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10753.068632] Lustre: lustre-OST0000: new disk, initializing [10753.079129] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10753.100713] Lustre: 327994:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10753.230498] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10758.751040] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10758.756738] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10758.879192] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10759.810896] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [10768.281406] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10784.749660] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10784.751288] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10784.761581] LustreError: Skipped 1 previous similar message [10784.767705] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10784.776723] Lustre: Skipped 1 previous similar message [10784.789956] Lustre: Skipped 1 previous similar message [10786.648055] Lustre: server umount lustre-OST0000 complete [10793.442792] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10793.456135] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10793.480579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10794.979909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10794.983234] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10797.296395] Lustre: server umount lustre-MDT0000 complete [10801.752937] LustreError: 327993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787572764 with bad export cookie 16388908796680555302 [10801.756962] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10801.773439] LustreError: 327993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10802.281688] Lustre: server umount lustre-MDT0001 complete [10813.958356] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing unload_modules_local [10816.637183] Key type lgssc unregistered [10816.958472] LNet: 329904:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10816.965377] LNetError: 329904:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10816.979467] LNet: Removed LNI 192.168.206.154@tcp [10817.732826] Key type .llcrypt unregistered [10817.734607] Key type ._llcrypt unregistered [10837.410781] Key type ._llcrypt registered [10837.416707] Key type .llcrypt registered [10837.541906] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [10838.890326] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10838.934071] alg: No test for adler32 (adler32-zlib) [10840.059640] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [10840.387177] LNet: Added LNI 192.168.206.154@tcp [8/256/0/180] [10842.088430] Key type lgssc registered [10843.234470] Lustre: Echo OBD driver; http://www.lustre.org/ [10851.634234] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10851.663242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10853.243370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10858.152790] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [10862.562707] LustreError: 331183:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10862.595475] LustreError: 331183:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10866.556131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10866.938711] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10871.394726] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [10878.593028] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10883.963633] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10890.475604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10890.826764] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10896.930363] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [10904.024246] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10913.100986] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10914.640893] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10918.369343] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10918.379400] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10918.394787] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10918.882783] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10918.903191] Lustre: Skipped 1 previous similar message [10918.906262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10918.909798] Lustre: Skipped 1 previous similar message [10922.079554] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.54@tcp (stopping) [10923.930150] Lustre: server umount lustre-MDT0000 complete [10924.002877] LustreError: 331182:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10924.021890] LustreError: 331182:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10927.187672] LustreError: 331178:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10927.895477] LustreError: 331161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787572890 with bad export cookie 14809473107104525120 [10927.903143] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10927.908397] LustreError: 331161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10929.124080] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10929.141857] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10932.129029] LustreError: 333643:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10932.145147] LustreError: 333643:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10934.181435] Lustre: server umount lustre-MDT0001 complete [10937.611429] Lustre: server umount lustre-OST0000 complete [10945.022787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10949.338722] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [10951.724852] Lustre: 335373:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.154@tcp [10953.515749] Lustre: server umount MGS complete [10960.958979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10961.324628] LustreError: 335944:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10961.334244] LustreError: 335944:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10961.397562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10965.808567] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [10971.625109] LustreError: 335945:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10971.653316] LustreError: 335945:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10973.707542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10974.173267] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10979.251735] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [10987.776997] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10988.124797] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10994.215744] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [11001.670884] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11003.090624] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11008.672571] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11011.197258] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 1 sec [11017.191194] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11017.212392] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11017.218899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11017.233364] Lustre: Skipped 3 previous similar messages [11021.488912] Lustre: server umount lustre-OST0000 complete [11025.378894] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11025.390115] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11025.405643] Lustre: Skipped 1 previous similar message [11025.411914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11031.086295] Lustre: server umount lustre-MDT0000 complete [11032.553796] LustreError: 335944:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11032.567554] LustreError: 335944:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11034.802664] LustreError: 335927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787572997 with bad export cookie 14809473107104525960 [11034.807778] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11034.811526] LustreError: 335927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11035.128335] Lustre: server umount lustre-MDT0001 complete [11045.767790] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing unload_modules_local [11048.395547] Key type lgssc unregistered [11048.661848] LNet: 339383:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11048.669368] LNetError: 339383:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11048.687958] LNet: Removed LNI 192.168.206.154@tcp [11049.557128] Key type .llcrypt unregistered [11049.561177] Key type ._llcrypt unregistered [11066.757872] Key type ._llcrypt registered [11066.763079] Key type .llcrypt registered [11066.883461] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_hostid [11081.780487] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [11082.863363] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11083.058423] alg: No test for adler32 (adler32-zlib) [11084.145472] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [11084.475593] LNet: Added LNI 192.168.206.154@tcp [8/256/0/180] [11086.144166] Key type lgssc registered [11087.126041] Lustre: Echo OBD driver; http://www.lustre.org/ [11118.469924] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11118.508680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11119.789707] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11119.846269] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11119.950700] Lustre: lustre-MDT0000: new disk, initializing [11120.011150] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11120.022172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11124.673294] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [11134.587116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11134.699875] Lustre: 342543:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11134.776646] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11134.782571] Lustre: Skipped 1 previous similar message [11134.884881] Lustre: lustre-MDT0001: new disk, initializing [11134.963578] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11134.998478] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11135.010222] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11138.989073] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [11146.679895] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11151.696625] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11157.997366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11158.265319] Lustre: lustre-OST0000: new disk, initializing [11158.269862] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11158.276892] Lustre: 344512:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11158.366285] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11164.436860] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [11168.291822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11168.303067] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11168.430338] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11171.999233] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11175.851080] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11176.002959] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11180.162149] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11180.418592] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11184.097977] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11184.098442] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11184.106058] LustreError: Skipped 1 previous similar message [11184.111310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11184.124426] Lustre: Skipped 1 previous similar message [11184.146057] Lustre: Skipped 1 previous similar message [11188.655946] Lustre: server umount lustre-OST0000 complete [11194.345773] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11194.357207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11194.375857] Lustre: Skipped 1 previous similar message [11194.392165] Lustre: Skipped 1 previous similar message [11197.995086] Lustre: server umount lustre-MDT0000 complete [11201.301328] LustreError: 342537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787573164 with bad export cookie 10291794194247267149 [11201.306086] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11201.315577] LustreError: 342537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11201.740162] Lustre: server umount lustre-MDT0001 complete [11210.721866] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 08:06:10 (1787573170) [11227.419754] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_hostid [11238.566227] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [11272.621336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11272.880547] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11272.914816] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11273.016534] Lustre: lustre-MDT0000: new disk, initializing [11273.179689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11273.226585] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11278.361087] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [11288.191744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11288.358242] Lustre: 349268:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11288.432853] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11288.439677] Lustre: Skipped 1 previous similar message [11288.446749] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [11288.518638] Lustre: lustre-MDT0001: new disk, initializing [11288.594940] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11288.624568] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11288.645411] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11293.644571] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [11302.091746] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11307.154306] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11313.867517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11314.079875] Lustre: lustre-OST0000: new disk, initializing [11314.087617] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11314.098631] Lustre: 351236:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11314.203029] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11320.263575] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [11324.003973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11324.013470] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11324.143378] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11329.551380] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11365.354561] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11365.357096] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11365.362667] LustreError: Skipped 1 previous similar message [11365.371248] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11365.380866] Lustre: Skipped 1 previous similar message [11369.400596] Lustre: server umount lustre-OST0000 complete [11375.592522] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11375.602263] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11375.604296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11375.618309] Lustre: Skipped 1 previous similar message [11375.631038] Lustre: Skipped 2 previous similar messages [11378.912323] Lustre: server umount lustre-MDT0000 complete [11380.713053] LustreError: 349277:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11380.738022] LustreError: 349277:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11382.458124] LustreError: 349260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787573345 with bad export cookie 10291794194247268031 [11382.464575] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11382.479877] LustreError: 349260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11382.968612] Lustre: server umount lustre-MDT0001 complete [11391.336281] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing unload_modules_local [11394.204970] Key type lgssc unregistered [11394.480710] LNet: 353410:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11394.485451] LNetError: 353410:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11395.496726] LNet: Removed LNI 192.168.206.154@tcp [11396.456506] Key type .llcrypt unregistered [11396.459154] Key type ._llcrypt unregistered [11479.279735] Key type ._llcrypt registered [11479.283180] Key type .llcrypt registered [11479.380869] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing unload_modules_local [11479.714303] Key type .llcrypt unregistered [11479.719298] Key type ._llcrypt unregistered [11539.086736] Key type ._llcrypt registered [11539.089832] Key type .llcrypt registered [11539.169795] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing unload_modules_local [11539.427275] Key type .llcrypt unregistered [11539.433644] Key type ._llcrypt unregistered [11564.340197] Key type ._llcrypt registered [11564.342249] Key type .llcrypt registered [11564.436495] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing unload_modules_local [11564.716507] Key type .llcrypt unregistered [11564.722232] Key type ._llcrypt unregistered [11593.313819] Key type ._llcrypt registered [11593.316596] Key type .llcrypt registered [11593.481910] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing unload_modules_local [11593.834274] Key type .llcrypt unregistered [11593.848087] Key type ._llcrypt unregistered [11661.991198] Key type ._llcrypt registered [11661.994450] Key type .llcrypt registered [11662.131757] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing unload_modules_local [11662.523269] Key type .llcrypt unregistered [11662.527673] Key type ._llcrypt unregistered [11688.991270] Key type ._llcrypt registered [11688.992939] Key type .llcrypt registered [11689.138713] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing unload_modules_local [11689.420469] Key type .llcrypt unregistered [11689.423961] Key type ._llcrypt unregistered [11714.072789] Key type ._llcrypt registered [11714.074974] Key type .llcrypt registered [11714.168127] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing unload_modules_local [11714.412507] Key type .llcrypt unregistered [11714.417061] Key type ._llcrypt unregistered [11739.015710] Key type ._llcrypt registered [11739.019351] Key type .llcrypt registered [11739.131860] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing unload_modules_local [11739.397472] Key type .llcrypt unregistered [11739.403202] Key type ._llcrypt unregistered [11772.355568] Key type ._llcrypt registered [11772.357431] Key type .llcrypt registered [11772.433222] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing unload_modules_local [11772.698078] Key type .llcrypt unregistered [11772.702963] Key type ._llcrypt unregistered [11794.975138] Key type ._llcrypt registered [11794.976781] Key type .llcrypt registered [11795.027626] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing unload_modules_local [11795.260746] Key type .llcrypt unregistered [11795.263383] Key type ._llcrypt unregistered [11853.168067] Key type ._llcrypt registered [11853.169917] Key type .llcrypt registered [11853.351869] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing unload_modules_local [11853.741648] Key type .llcrypt unregistered [11853.750108] Key type ._llcrypt unregistered [11882.025182] Key type ._llcrypt registered [11882.028096] Key type .llcrypt registered [11882.143334] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing unload_modules_local [11882.393323] Key type .llcrypt unregistered [11882.398378] Key type ._llcrypt unregistered [11908.770516] Key type ._llcrypt registered [11908.772057] Key type .llcrypt registered [11908.897914] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing unload_modules_local [11909.289205] Key type .llcrypt unregistered [11909.295315] Key type ._llcrypt unregistered [11955.489913] Key type ._llcrypt registered [11955.495512] Key type .llcrypt registered [11955.607544] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_hostid [11969.245734] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [11970.144648] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11970.173809] alg: No test for adler32 (adler32-zlib) [11971.191871] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [11971.354340] LNet: Added LNI 192.168.206.154@tcp [8/256/0/180] [11973.048535] Key type lgssc registered [11973.896708] Lustre: Echo OBD driver; http://www.lustre.org/ [12007.950583] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12007.994357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12009.192699] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [12009.231057] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [12009.295377] Lustre: tst250fs-MDT0000: new disk, initializing [12009.366739] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12009.409882] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12012.570073] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [12021.312725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12021.478151] Lustre: 373603:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12021.550741] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [12021.555682] Lustre: Skipped 1 previous similar message [12021.618881] Lustre: tst250fs-MDT0001: new disk, initializing [12021.687332] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12021.710427] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12021.714375] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12026.682492] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [12033.147432] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12041.216793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12041.504389] Lustre: tst250fs-OST0000: new disk, initializing [12041.516574] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [12041.536417] Lustre: 375236:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12041.645884] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12048.140671] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [12048.486472] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12048.501288] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12048.627507] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12057.722389] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12057.963896] Lustre: tst250fs-OST0001: new disk, initializing [12057.970217] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [12057.981740] Lustre: 376110:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12058.109705] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12063.260115] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12063.271658] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12063.300832] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000400 [12063.583000] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [12069.134225] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [12071.208284] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [12071.224559] Lustre: Skipped 1 previous similar message [12090.855471] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12090.871063] Lustre: Skipped 1 previous similar message [12090.874352] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12090.890462] Lustre: Skipped 3 previous similar messages [12095.414810] Lustre: server umount tst250fs-MDT0000 complete [12095.976757] LustreError: 373610:0:(ldlm_lib.c:1190:target_handle_connect()) tst250fs-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12096.010454] LustreError: 373610:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [12098.462943] LustreError: 374408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787574061 with bad export cookie 10638212292223287715 [12098.468609] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12098.469296] LustreError: 374408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12098.701357] Lustre: server umount tst250fs-MDT0001 complete [12112.428964] Lustre: server umount tst250fs-OST0000 complete [12126.272642] Lustre: server umount tst250fs-OST0001 complete [12144.269122] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_hostid [12150.940285] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [12179.195472] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 08:22:18 (1787574138) [12180.540355] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [12196.532413] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_hostid [12205.885576] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [12232.991296] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 11950 sec ========= 08:23:13 (1787574193) [12234.514830] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 08:23:14 (1787574194) === [12237.479472] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 08:23:17 (1787574197) === [12273.795173] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing unload_modules_local [12278.132338] Key type lgssc unregistered [12278.409496] LNet: 385250:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12278.424580] LNetError: 385250:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12278.453684] LNet: Removed LNI 192.168.206.154@tcp [12279.349954] Key type .llcrypt unregistered [12279.353972] Key type ._llcrypt unregistered