[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 592860077 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003358] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005030] kvm-guest: setup PV IPIs [ 0.009000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.010027] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011014] pid_max: default: 32768 minimum: 301 [ 0.012174] LSM: Security Framework initializing [ 0.013053] Yama: becoming mindful. [ 0.014039] SELinux: Initializing. [ 0.015087] *** VALIDATE selinux *** [ 0.026521] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.032313] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.033198] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.035037] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.036136] *** VALIDATE tmpfs *** [ 0.037586] *** VALIDATE proc *** [ 0.039152] *** VALIDATE cgroup *** [ 0.040009] *** VALIDATE cgroup2 *** [ 0.042042] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.044181] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.045010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.046040] Spectre V2 : User space: Vulnerable [ 0.047011] Speculative Store Bypass: Vulnerable [ 0.050136] debug: unmapping init [mem 0xffffffffb9a59000-0xffffffffb9a60fff] [ 0.052174] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.054378] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.055023] ... version: 2 [ 0.056012] ... bit width: 48 [ 0.057010] ... generic registers: 4 [ 0.058010] ... value mask: 0000ffffffffffff [ 0.059012] ... max period: 00007fffffffffff [ 0.060016] ... fixed-purpose events: 3 [ 0.061011] ... event mask: 000000070000000f [ 0.062379] rcu: Hierarchical SRCU implementation. [ 0.064647] smp: Bringing up secondary CPUs ... [ 0.065644] x86: Booting SMP configuration: [ 0.066021] .... node #0, CPUs: #1 #2 #3 [ 0.072590] smp: Brought up 1 node, 4 CPUs [ 0.074011] smpboot: Max logical packages: 1 [ 0.075020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.106000] node 0 deferred pages initialised in 28ms [ 0.109126] devtmpfs: initialized [ 0.111814] x86/mm: Memory block size: 128MB [ 0.120420] gcov: version magic: 0x41383552 [ 0.124308] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.125070] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.126754] pinctrl core: initialized pinctrl subsystem [ 0.130569] [ 0.132020] ************************************************************* [ 0.137015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.145030] ** ** [ 0.148020] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.150013] ** ** [ 0.155030] ** This means that this kernel is built to expose internal ** [ 0.159029] ** IOMMU data structures, which may compromise security on ** [ 0.161012] ** your system. ** [ 0.163011] ** ** [ 0.165010] ** If you see this message and you are not debugging the ** [ 0.167009] ** kernel, report this immediately to your vendor! ** [ 0.169011] ** ** [ 0.171011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.173011] ************************************************************* [ 0.177833] NET: Registered protocol family 16 [ 0.181421] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.187077] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.190063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.198031] cpuidle: using governor menu [ 0.199792] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.202414] PCI: Using configuration type 1 for base access [ 0.209557] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.236493] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.241035] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.246261] cryptd: max_cpu_qlen set to 1000 [ 0.251795] ACPI: Added _OSI(Module Device) [ 0.254014] ACPI: Added _OSI(Processor Device) [ 0.257014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.261015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.276552] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.296980] ACPI: Interpreter enabled [ 0.303083] ACPI: PM: (supports S0 S3 S4 S5) [ 0.306027] ACPI: Using IOAPIC for interrupt routing [ 0.309398] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.315885] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.327134] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.330118] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.333029] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.338112] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.345000] acpiphp: Slot [2] registered [ 0.345000] acpiphp: Slot [5] registered [ 0.347130] acpiphp: Slot [6] registered [ 0.350092] acpiphp: Slot [7] registered [ 0.351087] acpiphp: Slot [8] registered [ 0.354140] acpiphp: Slot [9] registered [ 0.357270] acpiphp: Slot [10] registered [ 0.360299] acpiphp: Slot [3] registered [ 0.362209] acpiphp: Slot [4] registered [ 0.365298] acpiphp: Slot [11] registered [ 0.368114] acpiphp: Slot [12] registered [ 0.371357] acpiphp: Slot [13] registered [ 0.374186] acpiphp: Slot [14] registered [ 0.377297] acpiphp: Slot [15] registered [ 0.381295] acpiphp: Slot [16] registered [ 0.383159] acpiphp: Slot [17] registered [ 0.385185] acpiphp: Slot [18] registered [ 0.386086] acpiphp: Slot [19] registered [ 0.387090] acpiphp: Slot [20] registered [ 0.389099] acpiphp: Slot [21] registered [ 0.392119] acpiphp: Slot [22] registered [ 0.395240] acpiphp: Slot [23] registered [ 0.397122] acpiphp: Slot [24] registered [ 0.399132] acpiphp: Slot [25] registered [ 0.400101] acpiphp: Slot [26] registered [ 0.402098] acpiphp: Slot [27] registered [ 0.403123] acpiphp: Slot [28] registered [ 0.405188] acpiphp: Slot [29] registered [ 0.407096] acpiphp: Slot [30] registered [ 0.409121] acpiphp: Slot [31] registered [ 0.411108] PCI host bridge to bus 0000:00 [ 0.413032] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.414022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.417025] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.419074] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.422030] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.425031] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.427178] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.429368] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.432308] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.445010] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.452064] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.455017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.457020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.461019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.465105] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.467901] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.469033] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.471997] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.478014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.492018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.501013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.507688] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.516026] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.527018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.555018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.571699] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.584027] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.591027] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.621200] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.639328] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.651026] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.663026] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.686024] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.706272] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.716023] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.727375] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.753030] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.771847] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.780576] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.787024] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.808588] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.825638] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.835611] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.846026] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.868033] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.892000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.895000] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.897930] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.904485] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.906136] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.915555] iommu: Default domain type: Passthrough [ 0.922793] SCSI subsystem initialized [ 0.926257] ACPI: bus type USB registered [ 0.929297] usbcore: registered new interface driver usbfs [ 0.931086] usbcore: registered new interface driver hub [ 0.933327] usbcore: registered new device driver usb [ 0.936209] pps_core: LinuxPPS API ver. 1 registered [ 0.940014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.946092] PTP clock support registered [ 0.953290] EDAC MC: Ver: 3.0.0 [ 0.955375] PCI: Using ACPI for IRQ routing [ 0.963885] NetLabel: Initializing [ 0.967016] NetLabel: domain hash size = 128 [ 0.970022] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.973091] NetLabel: unlabeled traffic allowed by default [ 0.975851] vgaarb: loaded [ 0.979586] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.983020] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.992499] clocksource: Switched to clocksource kvm-clock [ 1.261373] VFS: Disk quotas dquot_6.6.0 [ 1.264122] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.267934] *** VALIDATE ramfs *** [ 1.269492] *** VALIDATE hugetlbfs *** [ 1.274922] pnp: PnP ACPI init [ 1.283889] pnp: PnP ACPI: found 6 devices [ 1.340555] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.355513] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.364213] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.370383] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.378385] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.383657] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.387816] NET: Registered protocol family 2 [ 1.391054] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.395386] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.398182] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.403823] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.408476] TCP: Hash tables configured (established 65536 bind 65536) [ 1.411048] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.414086] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.416789] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.420429] NET: Registered protocol family 1 [ 1.422331] RPC: Registered named UNIX socket transport module. [ 1.424375] RPC: Registered udp transport module. [ 1.425734] RPC: Registered tcp transport module. [ 1.427390] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.429783] NET: Registered protocol family 44 [ 1.431223] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.432573] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.434476] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.437199] PCI: CLS 0 bytes, default 64 [ 1.439541] Unpacking initramfs... [ 3.613799] debug: unmapping init [mem 0xffff92f4bcc54000-0xffff92f4bffbffff] [ 3.620174] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.625293] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.635236] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 4.296926] Initialise system trusted keyrings [ 4.299114] Key type blacklist registered [ 4.302127] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.314067] zbud: loaded [ 4.319356] *** VALIDATE nfs *** [ 4.320847] *** VALIDATE nfs4 *** [ 4.323460] pstore: using deflate compression [ 4.331812] Platform Keyring initialized [ 4.468648] NET: Registered protocol family 38 [ 4.470807] Key type asymmetric registered [ 4.472307] Asymmetric key parser 'x509' registered [ 4.474289] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.478614] io scheduler mq-deadline registered [ 4.483125] io scheduler kyber registered [ 4.485283] io scheduler bfq registered [ 4.487982] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.491955] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.496183] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.504225] ACPI: Power Button [PWRF] [ 4.524632] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.533184] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.584175] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.606352] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.631206] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.665536] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.696927] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.703511] Non-volatile memory driver v1.3 [ 4.705531] Linux agpgart interface v0.103 [ 4.742888] virtio_blk virtio1: [vda] 145872 512-byte logical blocks (74.7 MB/71.2 MiB) [ 4.746931] vda: detected capacity change from 0 to 74686464 [ 4.784317] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.789110] vdb: detected capacity change from 0 to 1073741824 [ 4.842831] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.854486] vdc: detected capacity change from 0 to 2621440000 [ 4.898376] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.904813] vdd: detected capacity change from 0 to 2621440000 [ 4.943376] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.951595] vde: detected capacity change from 0 to 4294967296 [ 4.992372] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.009526] vdf: detected capacity change from 0 to 4294967296 [ 5.028666] libphy: Fixed MDIO Bus: probed [ 5.040502] usbcore: registered new interface driver usbserial_generic [ 5.046354] usbserial: USB Serial support registered for generic [ 5.052786] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.062920] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.068440] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.073058] mousedev: PS/2 mouse device common for all mice [ 5.078562] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.084532] rtc_cmos 00:05: RTC can wake from S4 [ 5.087765] rtc_cmos 00:05: registered as rtc0 [ 5.095097] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.100318] intel_pstate: CPU model not supported [ 5.108603] hid: raw HID events driver (C) Jiri Kosina [ 5.112062] usbcore: registered new interface driver usbhid [ 5.115201] usbhid: USB HID core driver [ 5.117426] drop_monitor: Initializing network drop monitor service [ 5.120850] Initializing XFRM netlink socket [ 5.123556] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.125276] NET: Registered protocol family 10 [ 5.132192] Segment Routing with IPv6 [ 5.139308] NET: Registered protocol family 17 [ 5.144370] mpls_gso: MPLS GSO support [ 5.151469] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.187480] RAS: Correctable Errors collector initialized. [ 5.190988] AVX version of gcm_enc/dec engaged. [ 5.196366] AES CTR mode by8 optimization enabled [ 5.314756] sched_clock: Marking stable (5314736211, 0)->(6394482333, -1079746122) [ 5.324776] registered taskstats version 1 [ 5.327215] Loading compiled-in X.509 certificates [ 5.330167] zswap: loaded using pool lzo/zbud [ 5.363725] Key type big_key registered [ 5.381613] Key type encrypted registered [ 5.383556] ima: No TPM chip found, activating TPM-bypass! [ 5.386355] ima: Allocated hash algorithm: sha1 [ 5.388539] ima: No architecture policies found [ 5.391242] evm: Initialising EVM extended attributes: [ 5.395238] evm: security.selinux [ 5.397174] evm: security.ima [ 5.398586] evm: security.capability [ 5.401303] evm: HMAC attrs: 0x1 [ 5.407833] rtc_cmos 00:05: setting system clock to 2026-09-02 14:22:49 UTC (1788358969) [ 5.428024] debug: unmapping init [mem 0xffffffffbaa03000-0xffffffffbabfffff] [ 5.437859] debug: unmapping init [mem 0xffffffffb9782000-0xffffffffb9a58fff] [ 5.458267] Write protecting the kernel read-only data: 28672k [ 5.467071] debug: unmapping init [mem 0xffffffffb7e03000-0xffffffffb7ffffff] [ 5.472548] debug: unmapping init [mem 0xffffffffb8714000-0xffffffffb87fffff] [ 5.537569] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.551868] systemd[1]: Detected virtualization kvm. [ 5.559737] systemd[1]: Detected architecture x86-64. [ 5.564386] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.602968] systemd[1]: No hostname configured. [ 5.607875] systemd[1]: Set hostname to . [ 5.614391] random: systemd: uninitialized urandom read (16 bytes read) [ 5.618796] systemd[1]: Initializing machine ID from random generator. [ 5.778713] random: ln: uninitialized urandom read (6 bytes read) [ 5.958418] random: systemd: uninitialized urandom read (16 bytes read) [ 5.963161] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 5.975751] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 5.996892] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.765651] device-mapper: uevent: version 1.0.3 [ 7.774602] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 9.988915] virtio_net virtio0 ens2: renamed from eth0 [ 10.029515] random: fast init done [ 10.719597] scsi host0: ata_piix [ 10.818718] scsi host1: ata_piix [ 10.823409] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 10.834648] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 14.923308] random: crng init done [ 14.925536] random: 7 urandom warning(s) missed due to ratelimiting [ 18.204841] dracut-initqueue[591]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 19.453272] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Sockets. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 22.421885] printk: systemd: 26 output lines suppressed due to ratelimiting [ 23.155522] SELinux: Disabled at runtime. [ 23.292579] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 23.310879] systemd[1]: Detected virtualization kvm. [ 23.320586] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 24.809950] systemd[1]: initrd-switch-root.service: Succeeded. [ 24.817581] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 24.839043] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 24.843963] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 24.850292] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 24.859798] systemd[1]: Starting Journal Service... Starting Journal Service... [ 24.874935] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... Starting Remount Root and Kernel File Systems... Mounting Huge Pages File System... [ OK ] Created slice User and Session Slice. [ OK ] Stopped target Switch Root. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Apply Kernel Variables... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting POSIX Message Queue File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. Starting udev Coldplug all Devices... [ 25.384396] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target Slices. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target rpc_pipefs.target. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 26.088406] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 26.873989] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 26.925689] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 27.208598] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 27.333892] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (6s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit) [*** ] A start job is running for Configur…-only root support (7s / no limit) [ *** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (8s / no limit) [ ***] A start job is running for Configur…-only root support (9s / no limit)[ 34.041660] Key type dns_resolver registered [ **] A start job is running for Configur…-only root support (9s / no limit)[ 34.963198] NFS: Registering the id_resolver key type [ 34.967823] Key type id_resolver registered [ 34.972980] Key type id_legacy registered [ *] A start job is running for Configur…only root support (10s / no limit) [ **] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg404-server login: [ 97.963500] hrtimer: interrupt took 9639071 ns [ 109.534429] libcfs: loading out-of-tree module taints kernel. [ 109.793894] Key type ._llcrypt registered [ 109.813914] Key type .llcrypt registered [ 110.142742] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_hostid [ 135.546410] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 137.455037] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 137.469605] alg: No test for adler32 (adler32-zlib) [ 139.328739] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 140.604639] LNet: Added LNI 192.168.204.104@tcp [8/256/0/180] [ 142.575549] Key type lgssc registered [ 144.228136] Lustre: Echo OBD driver; http://www.lustre.org/ [ 162.772831] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 200.258265] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 211.897821] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 211.910695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 213.171129] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 213.203612] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 213.290238] Lustre: lustre-MDT0000: new disk, initializing [ 213.411047] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 213.433963] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 217.412452] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 229.179276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 229.305806] Lustre: 6519:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 229.331614] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 229.334796] Lustre: Skipped 1 previous similar message [ 229.433195] Lustre: lustre-MDT0001: new disk, initializing [ 229.539438] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 229.573094] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 229.583537] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 233.776294] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 238.218909] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 246.836892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 247.041365] Lustre: lustre-OST0000: new disk, initializing [ 247.043961] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 247.047157] Lustre: 8456:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 247.087206] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 251.924143] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 251.935203] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 252.007926] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 252.689904] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 266.445596] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 266.591585] Lustre: lustre-OST0001: new disk, initializing [ 266.594786] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 266.601278] Lustre: 9528:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 266.687209] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 272.425146] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 275.523981] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 275.531322] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 275.598748] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 284.636894] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 294.042294] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 300.926374] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing check_logdir /tmp/testlogs/ [ 305.406260] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing yml_node [ 309.444192] Lustre: DEBUG MARKER: Client: 2.17.56.1 [ 311.762862] Lustre: DEBUG MARKER: MDS: 2.17.56.1 [ 314.045639] Lustre: DEBUG MARKER: OSS: 2.17.56.1 [ 315.741945] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Sep 2 10:27:57 EDT 2026 [ 330.804771] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 332.369938] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 347.619992] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 347.629842] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 347.647666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 352.223684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 352.228639] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 353.671822] Lustre: server umount lustre-MDT0000 complete [ 356.505837] LustreError: 6511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788359320 with bad export cookie 7386321160967543614 [ 356.509088] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 356.514078] LustreError: 6511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 357.345379] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 357.357996] Lustre: Skipped 2 previous similar messages [ 357.365172] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 357.367986] Lustre: Skipped 3 previous similar messages [ 361.378383] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 361.391060] Lustre: Skipped 1 previous similar message [ 362.847607] Lustre: server umount lustre-MDT0001 complete [ 366.488685] Lustre: server umount lustre-OST0000 complete [ 370.521896] Lustre: server umount lustre-OST0001 complete [ 377.275728] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_hostid [ 386.905800] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 410.886689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 411.041158] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 411.067389] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 411.112993] Lustre: lustre-MDT0000: new disk, initializing [ 411.166528] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 411.184797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 413.964522] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 420.210686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 420.269311] Lustre: 15075:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 420.307680] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 420.311203] Lustre: Skipped 1 previous similar message [ 420.360117] Lustre: lustre-MDT0001: new disk, initializing [ 420.413920] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 420.432612] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 420.439739] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 422.450605] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 428.582294] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 431.773861] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 435.829625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 436.056953] Lustre: 17046:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 437.789334] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 437.796788] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 437.902914] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 440.079944] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 446.611900] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 449.059635] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 449.198882] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 451.391260] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 451.517386] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 453.090506] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 453.090577] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 453.094098] LustreError: Skipped 1 previous similar message [ 453.100791] Lustre: Skipped 1 previous similar message [ 453.104357] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 453.109636] Lustre: Skipped 1 previous similar message [ 458.596655] Lustre: server umount lustre-OST0000 complete [ 461.282737] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 461.289480] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 461.300063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 461.303028] Lustre: Skipped 2 previous similar messages [ 466.542547] Lustre: server umount lustre-MDT0000 complete [ 468.450142] LustreError: 15089:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 468.462196] LustreError: 15089:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 468.495735] LustreError: 15070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788359432 with bad export cookie 7386321160967545147 [ 468.498609] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 468.501629] LustreError: 15070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 468.662632] Lustre: server umount lustre-MDT0001 complete [ 471.705371] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 10:30:34 (1788359434) [ 482.009600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 482.261144] LustreError: 19919:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 482.308635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 482.313596] Lustre: Skipped 1 previous similar message [ 484.639942] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 487.393148] LustreError: 19920:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 489.786503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 492.357578] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 496.965983] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 500.586973] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 501.752447] Lustre: Failing over lustre-MDT0000 [ 501.898996] Lustre: server umount lustre-MDT0000 complete [ 505.312678] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 505.324950] Lustre: Skipped 1 previous similar message [ 505.332577] LustreError: 19915:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 505.347155] LustreError: 19915:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 510.433987] LustreError: 20646:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 510.444122] LustreError: 20646:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 517.524711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 517.601479] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 517.755683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 517.758937] Lustre: Skipped 1 previous similar message [ 520.166541] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 523.236747] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 523.240937] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 523.253522] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 525.865543] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 532.448616] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 532.449340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 532.460969] Lustre: Skipped 2 previous similar messages [ 532.465957] Lustre: Skipped 2 previous similar messages [ 534.613759] Lustre: server umount lustre-MDT0000 complete [ 535.587965] LustreError: 19920:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 535.611176] LustreError: 19920:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 537.186470] LustreError: 19901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788359501 with bad export cookie 7386321160967546561 [ 537.191968] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 537.193668] LustreError: 19901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 537.389078] Lustre: server umount lustre-MDT0001 complete [ 542.896957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 545.955498] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 551.167647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 553.887162] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 558.369754] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 561.724429] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 562.917900] Lustre: Failing over lustre-MDT0000 [ 563.050739] Lustre: server umount lustre-MDT0000 complete [ 568.288516] LustreError: 23254:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 568.296017] LustreError: 23254:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 577.416793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 577.497774] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 579.872635] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 583.138911] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 583.141460] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 583.146523] Lustre: Skipped 1 previous similar message [ 583.159571] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 586.084109] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 592.354955] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 592.356280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 592.360038] Lustre: Skipped 3 previous similar messages [ 592.363534] Lustre: Skipped 3 previous similar messages [ 594.792754] Lustre: server umount lustre-MDT0000 complete [ 596.698938] LustreError: 23234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788359560 with bad export cookie 7386321160967547499 [ 596.701947] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 596.706288] LustreError: 23234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 596.865328] Lustre: server umount lustre-MDT0001 complete [ 601.703592] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 10:32:44 (1788359564) [ 611.198636] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_hostid [ 616.371982] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 637.889622] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 643.779463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 643.912936] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 643.935084] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 643.940262] Lustre: Skipped 1 previous similar message [ 643.986457] Lustre: lustre-MDT0000: new disk, initializing [ 643.988220] Lustre: Skipped 1 previous similar message [ 644.049977] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 644.053562] Lustre: Skipped 3 previous similar messages [ 644.062280] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 646.258233] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 652.388273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 652.447426] Lustre: 29617:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 652.598146] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 654.567135] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 659.679261] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 663.105520] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 666.726087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 666.973850] Lustre: 31587:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 668.331476] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 668.375437] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 668.385979] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 670.124639] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 675.701336] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 680.057056] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 691.665670] Lustre: server umount lustre-OST0000 complete [ 694.241270] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 694.243773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 694.249761] Lustre: Skipped 1 previous similar message [ 694.252841] Lustre: Skipped 1 previous similar message [ 703.492636] Lustre: server umount lustre-MDT0000 complete [ 704.480918] LustreError: 29628:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 704.490919] LustreError: 29628:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 705.357802] LustreError: 29608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788359669 with bad export cookie 7386321160967547905 [ 705.360491] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 705.365906] LustreError: 29608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 715.833356] Lustre: server umount lustre-MDT0001 complete [ 721.122763] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 722.568305] Key type lgssc unregistered [ 722.775641] LNet: 33540:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 722.782912] LNetError: 33540:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 722.803768] LNet: Removed LNI 192.168.204.104@tcp [ 723.358323] Key type .llcrypt unregistered [ 723.361288] Key type ._llcrypt unregistered [ 737.748085] Key type ._llcrypt registered [ 737.750046] Key type .llcrypt registered [ 737.799369] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 738.221308] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 738.275055] alg: No test for adler32 (adler32-zlib) [ 739.222410] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 739.360639] LNet: Added LNI 192.168.204.104@tcp [8/256/0/180] [ 740.984663] Key type lgssc registered [ 741.620454] Lustre: Echo OBD driver; http://www.lustre.org/ [ 745.919372] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 745.931870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 747.171123] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 749.411210] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 753.836656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 754.050107] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 756.340757] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 760.565545] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 763.253331] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 767.004426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 767.231180] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 770.183364] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 773.643152] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 779.502543] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 786.802160] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 10:35:49 (1788359749) [ 787.932347] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 788.993505] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 793.087367] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 10:35:55 (1788359755) [ 797.527377] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 10:36:00 (1788359760) [ 798.003857] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 801.996259] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 801.999897] Lustre: Skipped 2 previous similar messages [ 802.865739] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 10:36:05 (1788359765) [ 807.009639] Lustre: Setting parameter lustre-OST0000-osc.osc.max_pages_per_rpc=1024 in log lustre-client [ 807.013994] Lustre: Skipped 4 previous similar messages [ 815.229907] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 10:36:17 (1788359777) [ 816.772886] Lustre: *** cfs_fail_loc=131b, val=2*** [ 818.823567] Lustre: *** cfs_fail_loc=131b, val=2*** [ 823.297921] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 10:36:26 (1788359786) [ 824.166939] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 824.169872] Lustre: Skipped 9 previous similar messages [ 830.657920] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 10:36:33 (1788359793) [ 839.137557] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 839.141582] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 839.151398] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 841.185241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 841.189324] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 841.189599] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 843.998725] Lustre: server umount lustre-OST0000 complete [ 846.306448] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 846.306786] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 846.310183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 846.310193] Lustre: Skipped 1 previous similar message [ 846.331244] Lustre: Skipped 1 previous similar message [ 851.428688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 851.432376] Lustre: Skipped 2 previous similar messages [ 856.548227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 856.552369] Lustre: Skipped 1 previous similar message [ 860.127741] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 860.211169] Lustre: server umount lustre-MDT0000 complete [ 861.666710] LustreError: 35987:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 861.675422] LustreError: 35987:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 861.974747] LustreError: 34911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788359826 with bad export cookie 3020286421073072595 [ 861.978485] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 861.983224] LustreError: 34911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 862.156662] Lustre: server umount lustre-MDT0001 complete [ 866.970857] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 868.260587] Key type lgssc unregistered [ 868.422728] LNet: 42768:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 868.427925] LNetError: 42768:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 868.436506] LNet: Removed LNI 192.168.204.104@tcp [ 868.831664] Key type .llcrypt unregistered [ 868.833959] Key type ._llcrypt unregistered [ 877.060082] Key type ._llcrypt registered [ 877.061939] Key type .llcrypt registered [ 877.104868] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_hostid [ 885.678724] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 886.084760] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 886.137537] alg: No test for adler32 (adler32-zlib) [ 887.041821] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 887.175066] LNet: Added LNI 192.168.204.104@tcp [8/256/0/180] [ 888.775277] Key type lgssc registered [ 889.293437] Lustre: Echo OBD driver; http://www.lustre.org/ [ 906.679801] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 906.696154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 907.821165] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 907.836366] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 907.880125] Lustre: lustre-MDT0000: new disk, initializing [ 907.918074] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 907.930193] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 909.978164] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 915.886100] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 915.963589] Lustre: 45898:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 916.003547] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 916.006285] Lustre: Skipped 1 previous similar message [ 916.048556] Lustre: lustre-MDT0001: new disk, initializing [ 916.098518] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 916.122213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 916.128718] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 918.320408] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 923.407766] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 926.091981] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 929.441439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 929.588548] Lustre: lustre-OST0000: new disk, initializing [ 929.591141] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 929.595666] Lustre: 47864:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 929.641486] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 931.720401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 931.729215] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 931.787279] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 932.404412] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 937.771494] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 939.734897] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 939.832936] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 941.642774] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 941.731901] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 947.169495] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 947.171523] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 947.174560] LustreError: Skipped 1 previous similar message [ 947.179459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 947.184018] Lustre: Skipped 1 previous similar message [ 948.705110] Lustre: server umount lustre-OST0000 complete [ 952.288691] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 952.288775] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 952.290721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 952.303919] Lustre: Skipped 2 previous similar messages [ 956.720090] Lustre: server umount lustre-MDT0000 complete [ 958.430852] LustreError: 45889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788359922 with bad export cookie 13455770084101708259 [ 958.433597] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 958.437775] LustreError: 45889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 958.581481] Lustre: server umount lustre-MDT0001 complete [ 962.781893] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 10:38:45 (1788359925) [ 965.633851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 965.849555] LustreError: 50320:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 965.857217] LustreError: 50320:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 965.888781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 967.544266] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 971.232843] LustreError: 50321:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 971.524034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 971.703478] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 973.464299] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 976.538238] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 978.767508] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 981.677996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 981.811456] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 984.092696] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 987.181778] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 994.678851] LustreError: 52845:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 995.082408] Lustre: Modifying parameter general.timeout=1 in log params [ 996.311328] Lustre: Modifying parameter general.timeout=4 in log params [ 996.313987] Lustre: Skipped 2 previous similar messages [ 998.677869] Lustre: Modifying parameter general.timeout=10 in log params [ 998.681387] Lustre: Skipped 5 previous similar messages [ 1002.690422] Lustre: Modifying parameter general.timeout=20 in log params [ 1002.692941] Lustre: Skipped 9 previous similar messages [ 1010.955471] Lustre: Modifying parameter general.timeout=41 in log params [ 1010.959708] Lustre: Skipped 20 previous similar messages [ 1027.271875] Lustre: Modifying parameter general.timeout=84 in log params [ 1027.274270] Lustre: Skipped 42 previous similar messages [ 1049.177545] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 10:40:12 (1788360012) [ 1050.347845] Lustre: 59752:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1051.812178] Lustre: 59948:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1051.817494] Lustre: 59948:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1054.198779] Lustre: 60242:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1054.203974] Lustre: 60242:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1059.607134] Lustre: Disabling parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 in log params [ 1059.612105] Lustre: Skipped 61 previous similar messages [ 1066.292233] Lustre: 61600:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1066.303674] LustreError: 61600:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1068.225353] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 10:40:30 (1788360030) [ 1074.144464] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1074.148267] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1074.156544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1074.160235] Lustre: Skipped 1 previous similar message [ 1077.218645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1077.221844] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1077.231044] Lustre: Skipped 1 previous similar message [ 1079.455350] Lustre: server umount lustre-MDT0000 complete [ 1081.221615] LustreError: 50300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788360045 with bad export cookie 13455770084101709141 [ 1081.223313] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1081.230149] LustreError: 50300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1081.378258] Lustre: server umount lustre-MDT0001 complete [ 1093.559148] Lustre: server umount lustre-OST0000 complete [ 1106.050256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1106.202763] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1106.338174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1108.136527] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 1112.046064] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1112.110819] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1112.127645] Lustre: 64123:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1114.131413] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 1117.171853] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1119.533585] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1122.412025] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1122.531344] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1125.193320] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 1128.366183] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1134.107264] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1134.112921] Lustre: Skipped 7 previous similar messages [ 1134.132389] Lustre: 66689:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1140.709608] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1140.710049] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1140.711272] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1140.711280] Lustre: Skipped 2 previous similar messages [ 1140.716334] Lustre: Skipped 1 previous similar message [ 1140.719835] LustreError: Skipped 1 previous similar message [ 1142.428566] Lustre: server umount lustre-OST0000 complete [ 1150.052645] Lustre: server umount lustre-MDT0000 complete [ 1150.945285] LustreError: 64133:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1150.953425] LustreError: 64133:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1151.533233] LustreError: 65983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788360115 with bad export cookie 13455770084101711535 [ 1151.536727] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1151.541041] LustreError: 65983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1151.672096] Lustre: server umount lustre-MDT0001 complete [ 1155.684175] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 1156.815426] Key type lgssc unregistered [ 1156.951384] LNet: 67895:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1156.954573] LNetError: 67895:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1156.962405] LNet: Removed LNI 192.168.204.104@tcp [ 1157.295174] Key type .llcrypt unregistered [ 1157.296389] Key type ._llcrypt unregistered [ 1168.210985] Key type ._llcrypt registered [ 1168.212523] Key type .llcrypt registered [ 1168.269159] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 1168.624890] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1168.654061] alg: No test for adler32 (adler32-zlib) [ 1169.535739] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 1169.656205] LNet: Added LNI 192.168.204.104@tcp [8/256/0/180] [ 1171.247168] Key type lgssc registered [ 1171.752863] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1175.002982] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1175.011257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1176.227185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1177.848283] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 1181.049475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1181.288483] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1182.855856] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 1185.733205] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1187.868622] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1190.463494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1190.727222] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1193.234247] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 1196.133661] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1204.857769] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1211.361173] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1211.362640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1211.369565] Lustre: Skipped 1 previous similar message [ 1214.676543] Lustre: server umount lustre-MDT0000 complete [ 1216.203489] LustreError: 69267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788360180 with bad export cookie 8067361915831278653 [ 1216.209355] LustreError: 69267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1216.209893] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1216.332766] Lustre: server umount lustre-MDT0001 complete [ 1228.336637] Lustre: server umount lustre-OST0000 complete [ 1239.488132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1239.615185] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1239.628265] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1239.743779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1241.332336] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 1244.917055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1244.964518] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1244.983906] Lustre: 74149:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1245.183243] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1246.886781] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 1249.812700] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1252.355790] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1255.418549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1255.525529] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1255.725780] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1258.457417] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 1261.545925] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1267.258789] Lustre: 76726:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1268.075419] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1274.849791] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1274.851105] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1274.866823] Lustre: Skipped 1 previous similar message [ 1274.870649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1274.876913] Lustre: Skipped 3 previous similar messages [ 1279.972846] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1279.975541] Lustre: Skipped 1 previous similar message [ 1284.063326] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1284.135862] Lustre: server umount lustre-OST0000 complete [ 1286.111875] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1286.115934] LustreError: Skipped 1 previous similar message [ 1286.119448] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1286.127921] Lustre: Skipped 1 previous similar message [ 1286.132296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1290.209885] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1290.213624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1290.222402] Lustre: Skipped 1 previous similar message [ 1291.942544] Lustre: server umount lustre-MDT0000 complete [ 1293.692630] LustreError: 74140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788360257 with bad export cookie 8067361915831279472 [ 1293.696264] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1293.701026] LustreError: 74140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1293.840958] Lustre: server umount lustre-MDT0001 complete [ 1298.284617] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 1299.529118] Key type lgssc unregistered [ 1299.677704] LNet: 77932:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1299.682214] LNetError: 77932:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1299.691097] LNet: Removed LNI 192.168.204.104@tcp [ 1300.078281] Key type .llcrypt unregistered [ 1300.080200] Key type ._llcrypt unregistered [ 1311.297350] Key type ._llcrypt registered [ 1311.300240] Key type .llcrypt registered [ 1311.347676] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 1311.734589] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1311.782013] alg: No test for adler32 (adler32-zlib) [ 1312.673414] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 1312.780334] LNet: Added LNI 192.168.204.104@tcp [8/256/0/180] [ 1314.375149] Key type lgssc registered [ 1314.831668] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1318.292854] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1318.304436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1319.546184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1321.465860] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 1325.663377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1325.961205] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1327.821894] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 1331.162518] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1333.681264] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1336.979084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1337.258379] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1340.175912] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 1343.675298] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1356.569025] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1357.389070] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1357.394366] Lustre: Skipped 1 previous similar message [ 1358.654155] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1358.657624] Lustre: Skipped 2 previous similar messages [ 1360.727826] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1360.730450] Lustre: Skipped 4 previous similar messages [ 1364.863558] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1364.867716] Lustre: Skipped 8 previous similar messages [ 1372.991713] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1372.996566] Lustre: Skipped 16 previous similar messages [ 1389.524447] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1389.527841] Lustre: Skipped 37 previous similar messages [ 1404.553793] Lustre: 87417:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1405.921206] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 10:46:08 (1788360368) [ 1406.767927] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1407.735368] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 10:46:10 (1788360370) [ 1411.040410] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1411.044192] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1411.049827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1412.065189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1412.066930] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1412.068328] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1415.328418] Lustre: server umount lustre-OST0000 complete [ 1417.185837] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1417.186523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1417.191658] Lustre: Skipped 1 previous similar message [ 1417.195931] Lustre: Skipped 1 previous similar message [ 1422.304203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1422.307199] Lustre: Skipped 1 previous similar message [ 1423.082913] Lustre: server umount lustre-MDT0000 complete [ 1424.794297] LustreError: 81175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788360388 with bad export cookie 17243551813303738298 [ 1424.798848] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1424.802502] LustreError: 81175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1424.946688] Lustre: server umount lustre-MDT0001 complete [ 1429.857682] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 1431.098364] Key type lgssc unregistered [ 1431.247552] LNet: 88665:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1431.250883] LNetError: 88665:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1431.259396] LNet: Removed LNI 192.168.204.104@tcp [ 1431.646276] Key type .llcrypt unregistered [ 1431.647789] Key type ._llcrypt unregistered [ 1433.415182] Key type ._llcrypt registered [ 1433.416619] Key type .llcrypt registered [ 1433.464422] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1435.321031] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules [ 1435.707464] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1435.769298] alg: No test for adler32 (adler32-zlib) [ 1436.661562] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 1436.776121] LNet: Added LNI 192.168.204.104@tcp [8/256/0/180] [ 1438.391182] Key type lgssc registered [ 1438.924115] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1439.436507] LustreError: 89818:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 1448.866383] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 1452.766248] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1460.664972] LustreError: 89818:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1460.676822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1461.923777] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1463.842727] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 1468.025810] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 10:47:10 (1788360430) [ 1469.980089] LustreError: 90874:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1469.985103] LustreError: 90874:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1469.989546] LustreError: 90874:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 0, failed: rc = -5 [ 1470.125652] Lustre: server umount lustre-MDT0000 complete [ 1475.838900] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 1477.047934] Key type lgssc unregistered [ 1477.190826] LNet: 92366:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1477.195934] LNetError: 92366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1477.205556] LNet: Removed LNI 192.168.204.104@tcp [ 1477.572587] Key type .llcrypt unregistered [ 1477.574707] Key type ._llcrypt unregistered [ 1485.501767] Key type ._llcrypt registered [ 1485.503369] Key type .llcrypt registered [ 1485.545218] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 1485.888205] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1485.927182] alg: No test for adler32 (adler32-zlib) [ 1486.789161] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 1486.893253] LNet: Added LNI 192.168.204.104@tcp [8/256/0/180] [ 1488.479234] Key type lgssc registered [ 1488.916088] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1492.245422] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1492.255541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1493.438952] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1495.069683] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 1498.818374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1499.181839] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1500.983464] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 1503.993199] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1506.294505] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1508.915539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1509.177407] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1511.338689] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 1513.905361] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1543.406972] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 10:48:26 (1788360506) [ 1544.110518] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 1544.878986] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 10:48:27 (1788360507) [ 1549.281211] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1549.283366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1549.286082] Lustre: Skipped 2 previous similar messages [ 1549.289096] Lustre: Skipped 1 previous similar message [ 1552.189914] Lustre: server umount lustre-MDT0000 complete [ 1553.890800] LustreError: 93544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788360517 with bad export cookie 1338410532619377331 [ 1553.894967] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1553.897205] LustreError: 93544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1554.033872] Lustre: server umount lustre-MDT0001 complete [ 1565.114760] Lustre: server umount lustre-OST0000 complete [ 1569.491077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1569.687672] LustreError: 97832:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1569.695530] LustreError: 97832:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1569.757383] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1571.252629] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 1574.381966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1574.711502] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1576.295363] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 1579.101915] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1581.283113] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1586.033912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1586.108589] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1586.111681] LustreError: 97820:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1586.121320] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1586.124314] LustreError: 99912:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 1586.126405] LustreError: 99912:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 1586.128290] LustreError: 99912:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 1586.156750] Lustre: server umount lustre-OST0000 complete [ 1586.158315] LustreError: 99912:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1589.378708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1589.452596] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1589.455922] LustreError: 97821:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1589.459420] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1589.463789] LustreError: 100375:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 1589.466993] LustreError: 100375:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 1589.469157] LustreError: 100375:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 1589.494646] Lustre: server umount lustre-OST0000 complete [ 1589.495926] LustreError: 100375:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1593.034812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1593.110648] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1593.113354] Lustre: Found index 0 for lustre-OST0000, updating log [ 1593.116673] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 1593.143777] Lustre: lustre-OST0000: new disk, initializing [ 1593.145880] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1593.149630] Lustre: 100903:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1593.265063] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1595.479754] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 1596.900342] Lustre: Failing over lustre-OST0000 [ 1596.966693] Lustre: server umount lustre-OST0000 complete [ 1599.457597] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1599.458579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1599.462902] Lustre: Skipped 1 previous similar message [ 1599.467083] Lustre: Skipped 2 previous similar messages [ 1600.480918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1600.483034] Lustre: Skipped 1 previous similar message [ 1605.600611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1605.603017] Lustre: Skipped 1 previous similar message [ 1608.701640] Lustre: server umount lustre-MDT0000 complete [ 1610.116342] LustreError: 97814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788360574 with bad export cookie 1338410532619380061 [ 1610.121098] LustreError: 97814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1610.121807] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1610.250723] Lustre: server umount lustre-MDT0001 complete [ 1613.367722] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 10:49:36 (1788360576) [ 1619.950554] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 1625.404598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1625.757954] LustreError: 103247:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1625.859463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1629.963720] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 1631.201672] LustreError: 103248:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1635.297596] LustreError: 103247:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1638.531210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1644.345378] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 1647.307828] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 1654.588141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1655.764713] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 1663.745596] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 1664.963932] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1664.977713] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 1665.008826] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 1670.121324] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 1670.149245] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 1670.210679] Lustre: lustre-OST0000: Recovery over after 0:06, of 1 clients 1 recovered and 0 were evicted. [ 1670.214322] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 1670.223418] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 1674.274974] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1674.416061] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 1674.505984] Lustre: lustre-OST0001: new disk, initializing [ 1674.510061] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1674.514706] Lustre: 105568:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1674.768168] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1674.772433] Lustre: Skipped 2 previous similar messages [ 1680.554578] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1680.573667] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1680.727105] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1682.979392] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 1689.737053] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1694.276098] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1700.838050] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 1700.846071] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1700.856378] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1704.313824] Lustre: server umount lustre-MDT0001 complete [ 1706.473940] LustreError: 103242:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1706.496220] LustreError: 103242:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1713.434088] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1713.536115] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1713.540116] Lustre: Found index 1 for lustre-MDT0001, updating log [ 1713.606810] Lustre: 103237:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1715.253071] LustreError: 107145:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 1715.260429] LustreError: 107145:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 1715.268542] LustreError: 103239:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 1715.328827] LustreError: 107150:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.104@tcp: cfg command failed: rc = -17 [ 1715.339419] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 1715.339419] [ 1717.863917] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 1719.798676] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1725.334264] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 10:51:27 (1788360687) [ 1737.129209] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 1748.151398] Lustre: 109179:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1764.216661] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1766.767518] Lustre: 110313:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1842.147170] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1842.153391] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1842.158236] Lustre: Skipped 2 previous similar messages [ 1842.163407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1842.170316] Lustre: Skipped 4 previous similar messages [ 1845.515051] Lustre: server umount lustre-MDT0000 complete [ 1847.776710] LustreError: 103242:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1847.785401] LustreError: 103242:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 1847.832207] LustreError: 105581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788360811 with bad export cookie 1338410532619380901 [ 1847.837350] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1847.840978] LustreError: 105581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1848.008201] Lustre: server umount lustre-MDT0001 complete [ 1861.179330] Lustre: server umount lustre-OST0000 complete [ 1874.414607] Lustre: server umount lustre-OST0001 complete [ 1878.376032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1884.949537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1890.179058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1896.614056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1910.219531] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 1917.720125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1917.742368] Lustre: lustre-MDT0000: reset Object Index mappings [ 1918.141385] LustreError: 114263:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1918.252510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1918.256811] Lustre: Skipped 1 previous similar message [ 1921.257058] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 1927.649215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1927.666977] Lustre: lustre-MDT0001: reset Object Index mappings [ 1931.628839] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 1933.934973] Lustre: 115410:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1938.945390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1943.663745] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 1944.559899] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:387 to 0x280000401:418) [ 1950.099861] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1952.644363] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:161 to 0x2c0000401:193) [ 1952.645848] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:195 to 0x280000400:225) [ 1952.646111] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 1956.123570] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 1962.392291] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1964.924559] Lustre: 117287:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2004.828930] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 10:56:07 (1788360967) [ 2034.668085] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2034.676085] Lustre: Skipped 5 previous similar messages [ 2034.679915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2034.688617] Lustre: Skipped 6 previous similar messages [ 2039.179584] Lustre: server umount lustre-MDT0000 complete [ 2039.779890] LustreError: 114263:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2039.799324] LustreError: 114263:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 2041.727568] LustreError: 116599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788361005 with bad export cookie 1338410532619427997 [ 2041.743626] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2041.744991] LustreError: 116599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2044.906067] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2044.915191] Lustre: Skipped 2 previous similar messages [ 2056.161701] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2056.432097] Lustre: server umount lustre-MDT0001 complete [ 2065.635418] Lustre: server umount lustre-OST0000 complete [ 2074.675704] Lustre: server umount lustre-OST0001 complete [ 2079.508679] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_hostid [ 2086.278709] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 2110.737639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2110.894450] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2110.926129] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2111.014335] Lustre: lustre-MDT0000: new disk, initializing [ 2111.075460] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2111.079694] Lustre: Skipped 3 previous similar messages [ 2111.099381] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2114.037248] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 2118.852723] Lustre: server umount lustre-MDT0000 complete [ 2126.070348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2126.295400] Lustre: Found index 0 for lustre-MDT0000, updating log [ 2126.309983] Lustre: 121551:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2126.321453] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 2129.626758] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 2133.061324] Lustre: server umount lustre-MDT0000 complete [ 2138.682250] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 10:58:21 (1788361101) [ 2139.809955] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 2141.107321] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 10:58:23 (1788361103) [ 2153.895388] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 2160.349964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2163.616666] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 2169.729559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2169.821934] Lustre: 123598:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2169.870173] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2169.873571] Lustre: Skipped 1 previous similar message [ 2169.937952] Lustre: lustre-MDT0001: new disk, initializing [ 2170.036100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2170.044202] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2173.130064] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 2176.890979] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2181.594474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2181.804973] Lustre: lustre-OST0000: new disk, initializing [ 2181.808571] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2181.813391] Lustre: 125175:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2183.193955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2183.201138] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2183.327558] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2185.886560] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 2192.960256] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2193.073156] Lustre: lustre-OST0001: new disk, initializing [ 2193.076118] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2193.080710] Lustre: 126046:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2194.442654] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2194.452708] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2194.511806] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2197.095834] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 2204.360717] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2206.524631] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2425.175255] Lustre: 132446:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 2425.179774] Lustre: 132446:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 2425.183819] Lustre: 132446:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 2425.187776] Lustre: 132446:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 2425.191617] Lustre: 132446:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 2425.195506] Lustre: 132446:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 2584.476968] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 11:05:46 (1788361546) [ 2600.928694] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2600.933840] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2600.953724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2600.957721] Lustre: Skipped 8 previous similar messages [ 2603.326107] Lustre: server umount lustre-MDT0000 complete [ 2605.630216] LustreError: 125174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788361569 with bad export cookie 1338410532619493860 [ 2605.633662] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2605.637962] LustreError: 125174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2607.073246] LustreError: 126853:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2607.084049] LustreError: 126853:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 2611.891097] Lustre: server umount lustre-MDT0001 complete [ 2620.006190] Lustre: server umount lustre-OST0000 complete [ 2636.255129] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2636.364166] Lustre: server umount lustre-OST0001 complete [ 2639.493272] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_hostid [ 2644.631236] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 2662.422741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2662.544787] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2662.568553] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2662.625318] Lustre: lustre-MDT0000: new disk, initializing [ 2662.676323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2662.679549] Lustre: Skipped 5 previous similar messages [ 2662.687157] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2664.970435] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 2671.381337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2671.445625] Lustre: 135473:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2671.595103] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2673.910518] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 2678.861911] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2681.916159] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2685.360828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2685.495228] Lustre: 137442:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2686.675818] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2686.680644] Lustre: Skipped 1 previous similar message [ 2686.683396] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2686.725937] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2688.187515] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 2693.276572] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2698.450944] Lustre: lustre-MDD0000: changelog on [ 2699.997738] Lustre: lustre-MDD0001: changelog on [ 3499.929835] Lustre: lustre-MDD0001: changelog off [ 3500.653083] Lustre: 138885:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 3501.008443] Lustre: lustre-MDD0000: changelog off [ 3506.145499] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3506.145574] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3506.153952] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3506.157190] Lustre: Skipped 6 previous similar messages [ 3506.160520] Lustre: Skipped 8 previous similar messages [ 3508.962812] Lustre: server umount lustre-OST0000 complete [ 3511.265311] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3511.265340] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3511.273851] LustreError: Skipped 1 previous similar message [ 3516.384642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3516.386868] Lustre: Skipped 4 previous similar messages [ 3516.487209] Lustre: server umount lustre-MDT0000 complete [ 3517.863305] LustreError: 136276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788362481 with bad export cookie 1338410532620303697 [ 3517.866752] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3517.870421] LustreError: 136276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3518.018550] Lustre: server umount lustre-MDT0001 complete [ 3522.157453] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 3523.216927] Key type lgssc unregistered [ 3523.364570] LNet: 146367:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3523.368682] LNetError: 146367:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3523.377156] LNet: Removed LNI 192.168.204.104@tcp [ 3523.724133] Key type .llcrypt unregistered [ 3523.725424] Key type ._llcrypt unregistered [ 3531.165328] Key type ._llcrypt registered [ 3531.166621] Key type .llcrypt registered [ 3531.212062] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_hostid [ 3537.801489] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 3538.119748] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3538.149056] alg: No test for adler32 (adler32-zlib) [ 3539.011708] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 3539.103318] LNet: Added LNI 192.168.204.104@tcp [8/256/0/180] [ 3540.687146] Key type lgssc registered [ 3541.109743] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3553.315867] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3553.322117] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3554.421371] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3554.436952] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3554.476357] Lustre: lustre-MDT0000: new disk, initializing [ 3554.506055] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3554.513471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3556.254256] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 3560.838622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3560.883442] Lustre: 149541:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3560.907495] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3560.910030] Lustre: Skipped 1 previous similar message [ 3560.946741] Lustre: lustre-MDT0001: new disk, initializing [ 3560.984273] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3561.000133] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3561.004482] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3562.494061] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 3566.710903] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3568.798821] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3571.296102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3571.409625] Lustre: lustre-OST0000: new disk, initializing [ 3571.412617] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3571.416767] Lustre: 151510:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3571.449208] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3572.091238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3572.094765] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3572.142075] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3573.668753] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 3577.679587] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3580.961530] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_module obdecho/obdecho [ 3581.610297] LustreError: 152515:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 3581.614232] LustreError: 152515:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 3582.221733] LustreError: 152611:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 3586.527830] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3586.530880] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3586.536170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3587.555571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3587.557157] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3587.558462] Lustre: Skipped 1 previous similar message [ 3587.562862] Lustre: Skipped 1 previous similar message [ 3591.765867] Lustre: server umount lustre-MDT0000 complete [ 3593.113019] LustreError: 149534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788362557 with bad export cookie 12284264783833898297 [ 3593.115303] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3593.117198] LustreError: 149534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3593.235082] Lustre: server umount lustre-MDT0001 complete [ 3604.957389] Lustre: server umount lustre-OST0000 complete [ 3612.177643] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_hostid [ 3615.678268] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 3629.041933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3629.141351] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3629.153057] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3629.190863] Lustre: lustre-MDT0000: new disk, initializing [ 3629.215626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3629.222832] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3630.705517] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 3635.047314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3635.087138] Lustre: 156315:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3635.111518] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3635.113411] Lustre: Skipped 1 previous similar message [ 3635.147509] Lustre: lustre-MDT0001: new disk, initializing [ 3635.178798] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3635.190120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3635.195110] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3636.583810] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 3640.437571] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3642.387151] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3644.698199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3644.801444] Lustre: lustre-OST0000: new disk, initializing [ 3644.803292] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3644.806204] Lustre: 158284:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3644.833302] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3646.957823] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 3650.860468] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3652.427302] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3652.599577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3652.602783] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3652.637498] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3653.528369] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3655.005256] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3655.073373] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3657.698795] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3657.699205] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3657.702063] LustreError: Skipped 1 previous similar message [ 3657.702805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3657.706779] Lustre: Skipped 1 previous similar message [ 3657.708351] Lustre: Skipped 1 previous similar message [ 3661.786503] Lustre: server umount lustre-OST0000 complete [ 3665.887899] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3665.890506] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3665.894660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3665.896512] Lustre: Skipped 1 previous similar message [ 3669.273869] Lustre: server umount lustre-MDT0000 complete [ 3670.631083] LustreError: 156310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788362634 with bad export cookie 12284264783833899256 [ 3670.634389] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3670.635521] LustreError: 156310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3670.751643] Lustre: server umount lustre-MDT0001 complete [ 3671.986153] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 11:23:54 (1788362634) [ 3672.531628] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 3673.177708] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 11:23:56 (1788362636) [ 3675.907379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3676.057876] LustreError: 160700:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3676.064743] LustreError: 160700:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3676.081137] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3677.422203] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 3680.155622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3681.663742] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 3684.091676] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3685.937306] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3688.425755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3690.606275] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 3693.249846] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3705.823761] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3705.826382] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3705.830164] Lustre: Skipped 1 previous similar message [ 3705.832137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3705.834472] Lustre: Skipped 2 previous similar messages [ 3711.897432] Lustre: server umount lustre-MDT0000 complete [ 3712.480370] LustreError: 160719:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3712.487506] LustreError: 160719:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3713.130126] LustreError: 160682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788362677 with bad export cookie 12284264783833900110 [ 3713.133301] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3713.135346] LustreError: 160682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3714.160563] LustreError: 163188:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3714.167628] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.4@tcp (stopping) [ 3714.170046] Lustre: Skipped 4 previous similar messages [ 3717.599888] LustreError: 160695:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3717.600153] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3717.611106] Lustre: Skipped 2 previous similar messages [ 3719.328407] Lustre: server umount lustre-MDT0001 complete [ 3721.975758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3730.221315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3730.469948] LustreError: 164560:0:(llog.c:1655:llog_backup()) MGC192.168.204.104@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 3730.480300] Lustre: 164560:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.104@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 3743.202743] LustreError: 164565:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3743.226492] LustreError: 164565:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3753.442787] LustreError: 164566:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3753.460934] LustreError: 164566:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 3771.507481] LustreError: 164567:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3771.509362] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.4@tcp (not set up) [ 3771.533885] LustreError: 164567:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [ 3771.571839] Lustre: Skipped 2 previous similar messages [ 3791.841729] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 3791.847633] Lustre: 164560:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.204.104@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 3803.616839] LustreError: 164567:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3803.644618] LustreError: 164567:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 21 previous similar messages [ 3832.307713] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3832.313978] Lustre: Skipped 2 previous similar messages [ 3833.319661] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3835.012984] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 3840.458424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3858.927670] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 3862.409606] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 3864.553182] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3866.754870] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3867.636749] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3870.835664] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3871.857884] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3885.025311] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3885.025920] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3885.028429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3885.031325] Lustre: Skipped 1 previous similar message [ 3889.661260] Lustre: server umount lustre-MDT0000 complete [ 3890.147403] LustreError: 164567:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3890.162277] LustreError: 164567:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 44 previous similar messages [ 3891.932920] LustreError: 160682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788362856 with bad export cookie 12284264783833906585 [ 3891.935211] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3891.941084] LustreError: 160682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3892.149738] Lustre: server umount lustre-MDT0001 complete [ 3904.681082] Lustre: server umount lustre-OST0000 complete [ 3916.229668] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_hostid [ 3922.753485] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 3946.572293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3946.734099] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3946.755368] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3946.813284] Lustre: lustre-MDT0000: new disk, initializing [ 3946.875688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3946.880244] Lustre: Skipped 1 previous similar message [ 3946.894966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3949.888506] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 3957.078912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3957.141686] Lustre: 169661:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3957.352229] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3960.006533] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 3966.027734] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3969.542165] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3973.684040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3973.873852] Lustre: 171627:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3977.622105] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 3980.818437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3980.826063] Lustre: Skipped 1 previous similar message [ 3980.832986] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3980.887763] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3983.993348] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3986.826349] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3986.950551] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3989.720409] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3989.850194] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3996.128376] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3996.132635] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3996.138677] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3996.147176] Lustre: Skipped 1 previous similar message [ 3996.155424] Lustre: Skipped 2 previous similar messages [ 4005.343096] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4005.447328] Lustre: server umount lustre-OST0000 complete [ 4022.239349] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4022.350857] Lustre: server umount lustre-MDT0000 complete [ 4025.172211] LustreError: 169652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788362989 with bad export cookie 12284264783833923378 [ 4025.175293] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4025.182402] LustreError: 169652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4025.499819] Lustre: server umount lustre-MDT0001 complete [ 4032.947476] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 11:29:54 (1788362994) [ 4038.444535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4038.766988] LustreError: 174041:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4042.293437] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 4049.514632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4052.847754] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 4058.174652] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4062.425866] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4067.346778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4071.474858] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 4076.596636] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4096.023823] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 11:30:58 (1788363058) [ 4100.576200] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4100.580415] LustreError: Skipped 2 previous similar messages [ 4100.582677] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4100.589077] Lustre: Skipped 2 previous similar messages [ 4105.370163] Lustre: server umount lustre-OST0000 complete [ 4114.745797] Lustre: server umount lustre-MDT0000 complete [ 4117.516899] LustreError: 174021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788363081 with bad export cookie 12284264783833924260 [ 4117.521071] LustreError: 174021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4117.521143] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4117.723531] Lustre: server umount lustre-MDT0001 complete [ 4126.384878] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 4128.543720] Key type lgssc unregistered [ 4128.744866] LNet: 177788:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4128.754626] LNetError: 177788:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4128.765836] LNet: Removed LNI 192.168.204.104@tcp [ 4129.403912] Key type .llcrypt unregistered [ 4129.406272] Key type ._llcrypt unregistered [ 4144.642210] Key type ._llcrypt registered [ 4144.644255] Key type .llcrypt registered [ 4144.739152] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 4145.441054] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4145.454450] alg: No test for adler32 (adler32-zlib) [ 4146.410970] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 4146.600532] LNet: Added LNI 192.168.204.104@tcp [8/256/0/180] [ 4148.223238] Key type lgssc registered [ 4149.123131] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4154.961912] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4154.978352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4170.529510] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4175.647569] LustreError: 179030:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.104@tcp: failed processing log, type 4: rc = -110 [ 4180.767608] LustreError: 179030:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 4180.788054] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 4180.802391] LustreError: MGC192.168.204.104@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 4180.810625] LustreError: 179030:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 4180.816598] LustreError: 179030:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 4180.821514] LustreError: 179030:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 4180.863599] Lustre: server umount lustre-OST0000 complete [ 4180.866562] LustreError: 179030:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 4188.518384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4189.873071] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4193.395494] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 4199.404693] LustreError: 179529:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4199.423114] LustreError: 179529:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4200.790868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4201.052758] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4204.745603] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 4211.211959] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4216.258445] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4221.531299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4221.698584] LustreError: 181345:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 4221.713369] Lustre: 181345:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.204.104@tcp: invalid local config log lustre-OST0000: rc = -22 [ 4221.810941] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4226.367916] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 4232.282839] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4234.138144] LustreError: 179549:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 4234.150771] LustreError: 179549:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 4236.772374] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4236.779494] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4236.793497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4239.329902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4239.336018] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4239.337918] Lustre: Skipped 1 previous similar message [ 4239.353622] Lustre: Skipped 1 previous similar message [ 4242.442159] Lustre: server umount lustre-MDT0000 complete [ 4244.450154] LustreError: 179530:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4244.475727] LustreError: 179530:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4245.798727] LustreError: 179508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788363209 with bad export cookie 11484225505000931319 [ 4245.801592] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4245.807213] LustreError: 179508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4246.130463] Lustre: server umount lustre-MDT0001 complete [ 4259.733726] Lustre: server umount lustre-OST0000 complete [ 4267.109586] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_hostid [ 4274.259641] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 4301.986707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4302.129435] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4302.149887] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4302.219880] Lustre: lustre-MDT0000: new disk, initializing [ 4302.296385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4302.309696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4305.596137] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 4314.006901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4314.118264] Lustre: 184867:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4314.184689] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4314.193176] Lustre: Skipped 1 previous similar message [ 4314.258845] Lustre: lustre-MDT0001: new disk, initializing [ 4314.338393] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4314.365692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4314.377041] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4317.565831] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 4324.162258] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4328.405521] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4333.591489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4333.795764] Lustre: lustre-OST0000: new disk, initializing [ 4333.799113] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4333.803957] Lustre: 186833:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4333.865077] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4338.034895] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 4339.231711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4339.240080] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4339.307130] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4344.737761] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4347.862135] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4348.021931] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4350.956165] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4351.098955] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4354.531486] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4354.534825] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4354.540610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4354.543389] LustreError: Skipped 1 previous similar message [ 4354.547689] Lustre: Skipped 2 previous similar messages [ 4358.614498] Lustre: server umount lustre-OST0000 complete [ 4364.769533] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4364.770566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4364.780934] Lustre: Skipped 2 previous similar messages [ 4367.409864] Lustre: server umount lustre-MDT0000 complete [ 4369.888209] LustreError: 184879:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4369.894792] LustreError: 184879:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4370.024923] LustreError: 184858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788363334 with bad export cookie 11484225505000932040 [ 4370.028433] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4370.035766] LustreError: 184858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4370.238925] Lustre: server umount lustre-MDT0001 complete [ 4376.123684] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 11:35:38 (1788363338) [ 4387.896949] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 4389.637486] Key type lgssc unregistered [ 4389.812990] LNet: 189776:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4389.820502] LNetError: 189776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4389.833219] LNet: Removed LNI 192.168.204.104@tcp [ 4390.367322] Key type .llcrypt unregistered [ 4390.369849] Key type ._llcrypt unregistered [ 4402.591344] Key type ._llcrypt registered [ 4402.594340] Key type .llcrypt registered [ 4402.683694] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 4403.294561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4403.328800] alg: No test for adler32 (adler32-zlib) [ 4404.262322] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 4404.442954] LNet: Added LNI 192.168.204.104@tcp [8/256/0/180] [ 4406.073159] Key type lgssc registered [ 4406.768768] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4411.697763] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4411.714293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4412.962935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4415.491936] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 4420.750868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4420.954187] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4423.264396] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 4427.698194] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4431.082909] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4435.762915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4435.892517] Lustre: *** cfs_fail_loc=131e, val=0*** [ 4435.895876] LustreError: 192833:0:(llog.c:1696:llog_backup()) MGC192.168.204.104@tcp: failed to backup log lustre-OST0000: rc = -28 [ 4435.901415] Lustre: 192833:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.204.104@tcp: can't backup local config lustre-OST0000: rc = -28 [ 4435.978551] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4439.243651] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 4443.235752] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4446.563371] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4446.564801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4446.570301] Lustre: Skipped 2 previous similar messages [ 4446.581217] Lustre: Skipped 2 previous similar messages [ 4450.681164] Lustre: server umount lustre-MDT0000 complete [ 4451.811331] LustreError: 190968:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4451.818767] LustreError: 190968:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4452.778335] LustreError: 190952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788363416 with bad export cookie 14199253641411230762 [ 4452.783561] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4452.788952] LustreError: 190952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4454.751503] LustreError: 191700:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000f340b137 x1875234906987776/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4454.848591] Lustre: server umount lustre-MDT0001 complete [ 4467.672422] Lustre: server umount lustre-OST0000 complete [ 4474.339251] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_hostid [ 4481.484436] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 4505.891780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4506.065136] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4506.084166] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4506.133633] Lustre: lustre-MDT0000: new disk, initializing [ 4506.182236] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4506.195925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4509.137963] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 4516.290462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4516.373319] Lustre: 196354:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4516.404520] Lustre: *** cfs_fail_loc=131e, val=0*** [ 4516.409029] Lustre: Skipped 1 previous similar message [ 4516.411330] LustreError: 197138:0:(llog.c:1696:llog_backup()) MGC192.168.204.104@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 4516.418695] LustreError: 197138:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 4516.423547] Lustre: 197138:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.104@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 4516.440111] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4516.443619] Lustre: Skipped 1 previous similar message [ 4516.502559] Lustre: lustre-MDT0001: new disk, initializing [ 4516.570659] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4516.609320] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4516.618752] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4519.328073] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 4525.734551] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4529.281359] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4533.315416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4533.503552] Lustre: *** cfs_fail_loc=131e, val=0*** [ 4533.506666] Lustre: Skipped 1 previous similar message [ 4533.509365] LustreError: 198300:0:(llog.c:1696:llog_backup()) MGC192.168.204.104@tcp: failed to backup log lustre-OST0000: rc = -28 [ 4533.515408] LustreError: 198300:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 4533.519486] Lustre: 198300:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.104@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 4533.525776] Lustre: 198300:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 4533.538815] Lustre: lustre-OST0000: new disk, initializing [ 4533.541367] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4533.545498] Lustre: 198324:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4533.604845] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4537.137428] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 4541.465093] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4541.471950] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4541.531484] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4543.264214] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4546.172901] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4546.320813] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4549.349840] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4549.504704] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4551.654132] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4551.655757] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4551.657833] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4551.665266] Lustre: Skipped 1 previous similar message [ 4556.770456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4556.777437] Lustre: Skipped 2 previous similar messages [ 4556.970087] Lustre: server umount lustre-OST0000 complete [ 4561.891148] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4561.892178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4561.904830] Lustre: Skipped 1 previous similar message [ 4565.384780] Lustre: server umount lustre-MDT0000 complete [ 4567.008812] LustreError: 196367:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4567.021160] LustreError: 196367:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4567.512289] LustreError: 197150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788363531 with bad export cookie 14199253641411231483 [ 4567.515254] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4567.518472] LustreError: 197150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4567.688850] Lustre: server umount lustre-MDT0001 complete [ 4573.314743] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 11:38:55 (1788363535) [ 4585.202858] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 4592.389632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4592.630924] LustreError: 201447:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4592.670449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4595.638581] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 4597.730603] LustreError: 201448:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4601.490623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4604.455141] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 4606.381056] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4610.704684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4614.727474] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 4620.710673] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4620.832275] Lustre: lustre-OST0001: new disk, initializing [ 4620.836644] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4620.841571] Lustre: 203754:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4624.699370] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 4630.041282] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4630.047949] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4630.094430] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4632.598953] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4636.495865] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4637.558277] Lustre: DEBUG MARKER: ADD OST3 [ 4640.812732] loop: module loaded [ 4640.868543] loop0: detected capacity change from 0 to 204800000 [ 4641.034732] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4641.041716] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4641.048638] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4644.677607] loop0: detected capacity change from 0 to 204800000 [ 4646.890438] Lustre: DEBUG MARKER: STOP OST3 [ 4648.075448] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4648.191850] Lustre: lustre-OST0003: new disk, initializing [ 4648.194888] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 4648.201029] Lustre: 205613:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4648.302552] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 4648.308788] Lustre: Skipped 3 previous similar messages [ 4650.237109] Lustre: *** cfs_fail_loc=2109, val=2*** [ 4650.310374] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 4650.315270] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 4650.340360] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 4650.495065] Lustre: Failing over lustre-OST0003 [ 4650.561503] Lustre: server umount lustre-OST0003 complete [ 4651.781459] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 4652.255094] LustreError: 205917:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 4655.254885] LustreError: 202933:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.204.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4655.602849] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4655.604467] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4655.624566] LustreError: Skipped 1 previous similar message [ 4657.479302] Lustre: DEBUG MARKER: START OST3 again [ 4661.444465] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4661.603342] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 4663.101749] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4663.594204] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4663.594447] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4664.553746] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 4680.859889] Lustre: Failing over lustre-OST0003 [ 4680.916513] Lustre: server umount lustre-OST0003 complete [ 4682.219820] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4682.222534] LustreError: 202934:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4682.227258] Lustre: Skipped 2 previous similar messages [ 4682.241538] LustreError: 202934:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 4697.569713] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4697.579957] Lustre: Skipped 2 previous similar messages [ 4697.582633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4697.587399] Lustre: Skipped 4 previous similar messages [ 4701.392602] Lustre: server umount lustre-MDT0000 complete [ 4702.690128] LustreError: 201447:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4702.700041] LustreError: 201447:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 4703.561660] LustreError: 201429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788363667 with bad export cookie 14199253641411232365 [ 4703.565415] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4703.569148] LustreError: 201429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4707.814500] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4709.781314] Lustre: server umount lustre-MDT0001 complete [ 4711.963690] Lustre: server umount lustre-OST0000 complete [ 4713.910423] Lustre: server umount lustre-OST0001 complete [ 4716.875165] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_hostid [ 4720.787561] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 4740.427154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4740.608325] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4740.634520] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4740.690430] Lustre: lustre-MDT0000: new disk, initializing [ 4740.780853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4740.786467] Lustre: Skipped 1 previous similar message [ 4740.796148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4742.967757] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 4748.523770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4748.582721] Lustre: 211051:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4748.724101] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4750.575398] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 4755.573823] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4758.373922] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4761.732683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4761.873568] Lustre: lustre-OST0000: new disk, initializing [ 4761.875789] Lustre: Skipped 1 previous similar message [ 4761.878869] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4761.883684] Lustre: Skipped 2 previous similar messages [ 4761.887904] Lustre: 213016:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4763.786640] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4763.791417] Lustre: Skipped 1 previous similar message [ 4763.793658] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4763.843563] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4764.757620] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 4769.591464] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4771.722595] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4771.823637] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4773.801290] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4773.898649] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4779.493186] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4779.493885] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4779.497032] LustreError: Skipped 1 previous similar message [ 4779.497935] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4779.503979] Lustre: Skipped 4 previous similar messages [ 4779.512872] Lustre: Skipped 4 previous similar messages [ 4789.216145] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4789.298019] Lustre: server umount lustre-OST0000 complete [ 4794.850533] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4797.507573] Lustre: server umount lustre-MDT0000 complete [ 4799.510489] LustreError: 211041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788363763 with bad export cookie 14199253641411234654 [ 4799.513850] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4799.516471] LustreError: 211041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4799.686085] Lustre: server umount lustre-MDT0001 complete [ 4801.636538] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 11:42:44 (1788363764) [ 4810.074297] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_hostid [ 4815.485965] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 4833.883155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4833.996726] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4834.017224] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4834.060965] Lustre: lustre-MDT0000: new disk, initializing [ 4834.102402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4836.006216] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 4841.502450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4841.564058] Lustre: 217621:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4841.696082] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4843.639230] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 4849.125214] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4851.942498] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4855.154407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4855.291221] Lustre: 219590:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4856.447174] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4858.301591] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 4863.458838] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4865.436598] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4865.528521] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4867.288488] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4867.376185] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4871.648294] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4871.651898] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4871.658983] Lustre: Skipped 2 previous similar messages [ 4871.661962] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4871.665945] Lustre: Skipped 4 previous similar messages [ 4874.334167] Lustre: server umount lustre-OST0000 complete [ 4882.160494] Lustre: server umount lustre-MDT0000 complete [ 4882.402428] LustreError: 218575:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4882.416854] LustreError: 218575:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4883.908277] LustreError: 217612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788363848 with bad export cookie 14199253641411235543 [ 4883.908893] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4883.916614] LustreError: 217612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4884.109779] Lustre: server umount lustre-MDT0001 complete [ 4888.242052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4888.492459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4888.496767] Lustre: Skipped 5 previous similar messages [ 4890.299555] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 4894.410931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4896.546908] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 4900.150376] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4902.728051] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4906.129731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4909.130421] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 4912.751724] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4928.993096] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4928.998134] LustreError: Skipped 1 previous similar message [ 4931.560103] Lustre: server umount lustre-OST0000 complete [ 4939.436502] Lustre: server umount lustre-MDT0000 complete [ 4941.093018] LustreError: 221784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788363905 with bad export cookie 14199253641411236425 [ 4941.097929] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4941.099208] LustreError: 221784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4941.238963] Lustre: server umount lustre-MDT0001 complete [ 5046.534174] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 5047.760581] Key type lgssc unregistered [ 5047.907564] LNet: 225483:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5047.910668] LNetError: 225483:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5047.918781] LNet: Removed LNI 192.168.204.104@tcp [ 5048.300516] Key type .llcrypt unregistered [ 5048.302111] Key type ._llcrypt unregistered [ 5059.496343] Key type ._llcrypt registered [ 5059.498169] Key type .llcrypt registered [ 5059.552470] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_hostid [ 5067.326330] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 5067.727360] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5067.784469] alg: No test for adler32 (adler32-zlib) [ 5068.656549] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 5068.774565] LNet: Added LNI 192.168.204.104@tcp [8/256/0/180] [ 5070.367229] Key type lgssc registered [ 5070.870988] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5088.043367] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5088.053137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5089.172982] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5089.192520] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5089.238886] Lustre: lustre-MDT0000: new disk, initializing [ 5089.293909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5089.310311] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5091.254432] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 5096.919331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5096.990300] Lustre: 228811:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5097.023397] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5097.028019] Lustre: Skipped 1 previous similar message [ 5097.071109] Lustre: lustre-MDT0001: new disk, initializing [ 5097.122220] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5097.142051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5097.148721] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5099.287282] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 5104.186488] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5106.855882] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5110.222300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5110.341783] Lustre: lustre-OST0000: new disk, initializing [ 5110.344605] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5110.349367] Lustre: 230779:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5110.391576] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5112.076884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5112.082511] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5112.132492] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5113.110945] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 5117.945767] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5119.925158] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5120.012603] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5121.947583] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5122.039831] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5127.650424] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5127.656305] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5127.670585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5127.675334] Lustre: Skipped 1 previous similar message [ 5132.771619] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5132.776865] Lustre: Skipped 1 previous similar message [ 5137.375205] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5137.439082] Lustre: server umount lustre-OST0000 complete [ 5143.009173] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5143.010383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5143.016040] Lustre: Skipped 2 previous similar messages [ 5145.315299] Lustre: server umount lustre-MDT0000 complete [ 5146.849163] LustreError: 230778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364110 with bad export cookie 2780614241791898479 [ 5146.852592] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5146.853759] LustreError: 230778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5146.980452] Lustre: server umount lustre-MDT0001 complete [ 5154.626162] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 5158.676218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5158.853476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5160.430968] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 5163.625619] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5163.768651] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5165.367001] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 5166.531273] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5169.076821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5171.446640] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 5174.761878] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5174.834409] Lustre: lustre-OST0001: new disk, initializing [ 5174.837367] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5174.843288] Lustre: 236014:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5174.883290] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5174.885928] Lustre: Skipped 1 previous similar message [ 5176.391662] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5176.395739] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5176.425515] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5177.089400] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 5182.934413] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5188.089749] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5192.689457] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 11:49:15 (1788364155) [ 5194.210048] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5194.212710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5194.217084] Lustre: Skipped 3 previous similar messages [ 5194.219303] Lustre: Skipped 3 previous similar messages [ 5199.330760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5199.334069] Lustre: Skipped 2 previous similar messages [ 5199.585814] Lustre: server umount lustre-MDT0000 complete [ 5200.990917] LustreError: 236842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364165 with bad export cookie 2780614241791899361 [ 5200.996498] LustreError: 236842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5200.996545] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5201.121504] Lustre: server umount lustre-MDT0001 complete [ 5212.685633] Lustre: server umount lustre-OST0000 complete [ 5224.626025] Lustre: server umount lustre-OST0001 complete [ 5227.072049] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_hostid [ 5230.892727] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 5245.334971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5245.431081] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5245.446060] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5245.489692] Lustre: lustre-MDT0000: new disk, initializing [ 5245.523454] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5245.532784] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5247.037396] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 5251.659700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5251.707150] Lustre: 239956:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5251.820162] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5253.324975] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 5257.206346] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5259.206461] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5261.620367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5261.722336] Lustre: lustre-OST0000: new disk, initializing [ 5261.723884] Lustre: Skipped 1 previous similar message [ 5261.725578] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5261.728947] Lustre: Skipped 2 previous similar messages [ 5261.731396] Lustre: 241923:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5263.543504] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5263.547478] Lustre: Skipped 1 previous similar message [ 5263.549733] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5263.576852] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5263.828527] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 5267.849291] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5269.345351] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5269.422809] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5270.859319] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5270.940559] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5274.084671] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5274.084934] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5274.085835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5274.090701] Lustre: Skipped 1 previous similar message [ 5274.093597] LustreError: Skipped 2 previous similar messages [ 5277.714073] Lustre: server umount lustre-OST0000 complete [ 5279.200738] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5279.204557] Lustre: Skipped 1 previous similar message [ 5285.144373] Lustre: server umount lustre-MDT0000 complete [ 5286.334078] LustreError: 239948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364250 with bad export cookie 2780614241791901454 [ 5286.337149] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5286.338119] LustreError: 239948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5286.450050] Lustre: server umount lustre-MDT0001 complete [ 5289.382199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5289.581036] LustreError: 244135:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5289.588331] LustreError: 244135:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5289.611800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5289.614967] Lustre: Skipped 2 previous similar messages [ 5291.235505] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 5294.573320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5296.425988] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 5299.067972] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5301.103642] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5303.639043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5306.021914] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 5308.711943] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5453.279856] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5453.282969] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5453.287087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5453.289548] Lustre: Skipped 5 previous similar messages [ 5457.380102] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5459.059731] Lustre: server umount lustre-OST0000 complete [ 5466.782072] Lustre: server umount lustre-MDT0000 complete [ 5467.616029] LustreError: 246628:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5467.622651] LustreError: 246628:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5468.278132] LustreError: 244117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364432 with bad export cookie 2780614241791902336 [ 5468.283190] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5468.284228] LustreError: 244117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5468.425650] Lustre: server umount lustre-MDT0001 complete [ 5574.420482] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 5575.536549] Key type lgssc unregistered [ 5575.675469] LNet: 247746:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5575.679339] LNetError: 247746:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5575.686521] LNet: Removed LNI 192.168.204.104@tcp [ 5576.026152] Key type .llcrypt unregistered [ 5576.027719] Key type ._llcrypt unregistered [ 5585.383416] Key type ._llcrypt registered [ 5585.384802] Key type .llcrypt registered [ 5585.424950] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_hostid [ 5592.322508] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 5592.753139] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5592.787757] alg: No test for adler32 (adler32-zlib) [ 5593.647323] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 5593.737928] LNet: Added LNI 192.168.204.104@tcp [8/256/0/180] [ 5595.319142] Key type lgssc registered [ 5595.668959] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5611.794785] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 5615.779883] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5615.789782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5616.908178] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5616.923313] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5616.972130] Lustre: lustre-MDT0000: new disk, initializing [ 5617.006447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5617.014505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5618.612557] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 5623.263038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5623.316032] Lustre: 251785:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5623.349270] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5623.352640] Lustre: Skipped 1 previous similar message [ 5623.391670] Lustre: lustre-MDT0001: new disk, initializing [ 5623.430111] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5623.450392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5623.457732] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5625.127923] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 5627.518665] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5629.886620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5630.011216] Lustre: lustre-OST0000: new disk, initializing [ 5630.013647] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5630.016965] Lustre: 253417:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5630.063527] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5632.326297] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 5635.585061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5635.589221] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5635.625761] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5637.135400] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5637.215179] Lustre: lustre-OST0001: new disk, initializing [ 5637.218447] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5637.224221] Lustre: 254286:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5637.270765] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5639.467434] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 5644.799137] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5644.803719] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5644.834339] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5646.162987] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5648.860264] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5654.496024] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5654.499111] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5654.507934] Lustre: server umount lustre-MDT0000 complete [ 5655.011240] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5655.016881] Lustre: Skipped 2 previous similar messages [ 5655.806894] LustreError: 252579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364619 with bad export cookie 11639557422146307213 [ 5655.807623] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5655.810963] LustreError: 252579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5655.816476] LustreError: 251789:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 5657.974858] Lustre: server umount lustre-MDT0001 complete [ 5669.388042] Lustre: server umount lustre-OST0000 complete [ 5675.999290] LustreError: 251789:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 5696.521169] Lustre: server umount lustre-OST0001 complete [ 5701.101854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5704.109049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5706.905540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5710.023074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5716.330072] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 5720.080341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5720.087457] Lustre: lustre-MDT0000: reset Object Index mappings [ 5720.232288] LustreError: 259271:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5720.240097] LustreError: 259271:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 5720.254798] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5721.693257] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 5724.645293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5724.652392] Lustre: lustre-MDT0001: reset Object Index mappings [ 5726.204259] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 5727.180563] Lustre: 260412:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5729.670479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5729.821719] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5729.824751] Lustre: Skipped 1 previous similar message [ 5730.850142] LustreError: 260766:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5730.854865] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5732.027740] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 5733.925614] LustreError: 260939:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5733.933260] LustreError: 260939:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5734.988976] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5737.047322] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 5740.331700] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5740.523091] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 5741.712549] Lustre: 262279:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5745.119886] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5745.122661] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5745.127943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5745.638313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5745.640225] Lustre: Skipped 2 previous similar messages [ 5749.213971] Lustre: server umount lustre-MDT0000 complete [ 5750.495280] LustreError: 259252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364714 with bad export cookie 11639557422146313828 [ 5750.497671] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5750.499383] LustreError: 259252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5750.622039] Lustre: server umount lustre-MDT0001 complete [ 5762.178761] Lustre: server umount lustre-OST0000 complete [ 5773.789787] Lustre: server umount lustre-OST0001 complete [ 5780.127053] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 11:59:02 (1788364742) [ 5785.511307] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_hostid [ 5788.999733] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 5802.284222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5802.366205] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5802.377585] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5802.411848] Lustre: lustre-MDT0000: new disk, initializing [ 5802.435661] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5802.437791] Lustre: Skipped 1 previous similar message [ 5802.444231] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5803.980360] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 5808.410323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5808.450766] Lustre: 266526:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5808.558890] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5809.908255] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 5813.700270] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5815.795181] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5818.328104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5818.456969] Lustre: lustre-OST0000: new disk, initializing [ 5818.458746] Lustre: Skipped 1 previous similar message [ 5818.460539] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5818.463993] Lustre: Skipped 2 previous similar messages [ 5818.466876] Lustre: 268493:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5820.027310] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5820.033179] Lustre: Skipped 1 previous similar message [ 5820.036246] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5820.073361] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5820.700677] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 5824.666943] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5826.187492] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5826.264973] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5827.696959] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5827.780566] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5830.626362] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5830.626435] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5830.627242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5830.638607] Lustre: Skipped 4 previous similar messages [ 5834.582279] Lustre: server umount lustre-OST0000 complete [ 5839.327907] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5839.331499] LustreError: Skipped 1 previous similar message [ 5839.333058] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5839.337796] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5839.340390] Lustre: Skipped 1 previous similar message [ 5842.137537] Lustre: server umount lustre-MDT0000 complete [ 5843.438642] LustreError: 266519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364807 with bad export cookie 11639557422146315900 [ 5843.442655] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5843.443412] LustreError: 266519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5843.566526] Lustre: server umount lustre-MDT0001 complete [ 5849.762815] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 5853.520267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5853.676683] LustreError: 271421:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5853.692264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5853.695531] Lustre: Skipped 2 previous similar messages [ 5855.093093] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 5858.253343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5860.159019] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 5861.256901] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5863.796873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5866.132377] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 5869.309525] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5869.376706] Lustre: lustre-OST0001: new disk, initializing [ 5869.378728] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5869.381602] Lustre: 273726:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5871.301536] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5871.304697] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5871.341249] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5871.525706] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 5876.239722] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5877.655944] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5878.518321] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 5879.717538] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 5879.722369] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 5883.874276] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5883.875570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5883.877971] Lustre: Skipped 3 previous similar messages [ 5883.880293] Lustre: Skipped 3 previous similar messages [ 5887.905234] Lustre: server umount lustre-MDT0000 complete [ 5888.992388] LustreError: 271422:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5888.997030] LustreError: 271422:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5889.153397] LustreError: 271404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364853 with bad export cookie 11639557422146316782 [ 5889.156818] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5889.158469] LustreError: 271404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5889.283173] Lustre: server umount lustre-MDT0001 complete [ 5900.298310] Lustre: server umount lustre-OST0000 complete [ 5911.903692] Lustre: server umount lustre-OST0001 complete [ 5917.723536] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 5921.300256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5921.451710] LustreError: 276789:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5921.467874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5921.469770] Lustre: Skipped 3 previous similar messages [ 5922.809830] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 5925.794268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5927.413922] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 5928.406333] Lustre: 277929:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5930.812114] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5932.009199] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:4 to 0x280000403:65) [ 5933.180128] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 5936.279452] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5937.392973] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 5938.443913] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 5941.880938] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5943.263840] Lustre: 279804:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5945.395148] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 5945.396687] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000403 to 0x280000404 [ 5948.863351] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 12:01:51 (1788364911) [ 5961.696163] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5961.699448] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5961.704301] Lustre: Skipped 1 previous similar message [ 5961.707217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5966.558710] Lustre: server umount lustre-MDT0000 complete [ 5968.012813] LustreError: 276772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364932 with bad export cookie 11639557422146323089 [ 5968.018211] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5968.018902] LustreError: 276772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5968.352400] LustreError: 278441:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5968.358805] LustreError: 278441:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5974.196608] Lustre: server umount lustre-MDT0001 complete [ 5981.630585] Lustre: server umount lustre-OST0000 complete [ 5989.019142] Lustre: server umount lustre-OST0001 complete [ 5991.128353] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_hostid [ 5994.357794] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 6009.733738] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 6013.547160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6013.638586] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6013.650724] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6013.690794] Lustre: lustre-MDT0000: new disk, initializing [ 6013.719792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6015.280507] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 6020.011876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6020.062931] Lustre: 284032:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6020.189431] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6021.773016] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 6026.844304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6026.958059] Lustre: 285665:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6027.228866] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6029.196399] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 6033.794054] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6035.187977] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6036.146974] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 6040.808304] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6046.494048] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 12:03:29 (1788365009) [ 6050.788473] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6050.790624] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6050.797413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6050.802608] Lustre: Skipped 7 previous similar messages [ 6050.808369] Lustre: Skipped 11 previous similar messages [ 6053.407780] Lustre: server umount lustre-MDT0000 complete [ 6054.691295] LustreError: 284839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788365018 with bad export cookie 11639557422146330957 [ 6054.694042] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6054.695229] LustreError: 284839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6054.819225] Lustre: server umount lustre-MDT0001 complete [ 6066.438793] Lustre: server umount lustre-OST0000 complete [ 6077.990309] Lustre: server umount lustre-OST0001 complete [ 6081.003621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6081.197958] LustreError: 288778:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6081.207657] LustreError: 288778:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6081.246711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6081.249440] Lustre: Skipped 7 previous similar messages [ 6082.745578] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 6086.234190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6088.152376] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 6091.223590] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6093.428438] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6096.146369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6098.566358] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 6101.341389] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6112.691533] loop1: detected capacity change from 0 to 204800000 [ 6112.765139] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6112.769568] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6112.773542] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6114.536532] loop1: detected capacity change from 0 to 204800000 [ 6116.550451] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6116.586300] LustreError: 288768:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs16728: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 6116.593327] LustreError: 288768:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs16728 [ 6116.596988] LustreError: 288768:0:(mgs_handler.c:517:mgs_target_reg()) fs16728-MDT0001: Error checking target index: rc = -13 [ 6116.602771] LustreError: fs16728-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 6116.606046] LustreError: 292493:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 6116.608368] LustreError: 292493:0:(tgt_mount.c:2129:server_put_super()) no obd fs16728-MDT0001 [ 6116.610656] LustreError: 292493:0:(tgt_mount.c:116:server_deregister_mount()) fs16728-MDT0001 not registered [ 6116.621955] Lustre: server umount fs16728-MDT0001 complete [ 6116.623773] LustreError: 292493:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 6119.599833] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6119.630817] Lustre: Setting parameter fs16728-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs16728-MDT0001 [ 6119.635369] Lustre: Skipped 2 previous similar messages [ 6119.656331] Lustre: srv-fs16728-MDT0001: No data found on store. Initialize space. [ 6119.658199] Lustre: Skipped 4 previous similar messages [ 6119.695924] Lustre: fs16728-MDT0001: new disk, initializing [ 6119.697541] Lustre: Skipped 3 previous similar messages [ 6119.738714] Lustre: srv-fs16728-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 6119.742357] Lustre: 292958:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs16728-MDT0001-mdtlov: can't create new llog: rc = -115 [ 6119.746422] LustreError: 292958:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs16728-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 6121.222692] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 6124.292256] Lustre: server umount fs16728-MDT0001 complete [ 6129.218024] loop1: detected capacity change from 0 to 204800000 [ 6131.220591] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6131.350373] Lustre: 294522:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs16728-MDT0001-mdtlov: can't create new llog: rc = -115 [ 6131.352885] LustreError: 294522:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs16728-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 6132.794859] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 6134.577211] Lustre: server umount fs16728-MDT0001 complete [ 6142.433646] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6142.435097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6142.438053] Lustre: Skipped 3 previous similar messages [ 6142.440321] Lustre: Skipped 1 previous similar message [ 6144.346074] Lustre: server umount lustre-MDT0000 complete [ 6147.237135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6147.281311] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6147.344547] LustreError: 290608:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6147.351810] LustreError: 290608:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 25 previous similar messages [ 6148.811386] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 6151.533460] loop1: detected capacity change from 0 to 204800000 [ 6152.680048] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6153.409144] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6153.518952] Lustre: 296965:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs16728-MDT0001-mdtlov: can't create new llog: rc = -115 [ 6153.522909] LustreError: 296965:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs16728-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 6154.956270] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 6156.647707] Lustre: server umount fs16728-MDT0001 complete [ 6168.041195] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6170.198616] Lustre: server umount lustre-OST0000 complete [ 6177.819883] Lustre: server umount lustre-MDT0000 complete [ 6179.226280] LustreError: 297456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788365143 with bad export cookie 11639557422146333631 [ 6179.230486] LustreError: 297456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6179.361538] Lustre: server umount lustre-MDT0001 complete [ 6183.573673] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 6184.703437] Key type lgssc unregistered [ 6184.836545] LNet: 299304:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6184.839937] LNetError: 299304:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6184.850349] LNet: Removed LNI 192.168.204.104@tcp [ 6185.179209] Key type .llcrypt unregistered [ 6185.181061] Key type ._llcrypt unregistered [ 6193.288269] Key type ._llcrypt registered [ 6193.290699] Key type .llcrypt registered [ 6193.331678] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 6193.772186] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6193.795242] alg: No test for adler32 (adler32-zlib) [ 6194.656360] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 6194.755536] LNet: Added LNI 192.168.204.104@tcp [8/256/0/180] [ 6196.351187] Key type lgssc registered [ 6196.769296] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6199.724671] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6199.735315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6200.896535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6202.403232] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 6205.485688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6205.659594] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6207.230156] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 6209.687547] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6211.655180] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6214.071793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6214.201274] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6215.205880] LustreError: 302381:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6215.211816] LustreError: 302381:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6216.199429] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 6218.882799] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6220.258056] LustreError: 302383:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6225.380395] LustreError: 302381:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6225.386452] LustreError: 302381:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6229.625479] LustreError: 302561:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6234.743626] LustreError: 302382:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6234.753890] LustreError: 302382:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6236.088636] loop1: detected capacity change from 0 to 204800000 [ 6236.210203] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6236.214128] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6236.218412] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6237.990407] loop1: detected capacity change from 0 to 204800000 [ 6240.024445] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6240.084321] Lustre: fs21688-OST0000: new disk, initializing [ 6240.087316] Lustre: srv-fs21688-OST0000: No data found on store. Initialize space. [ 6240.092289] Lustre: 304305:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6240.119872] Lustre: fs21688-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6242.173086] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 6244.977539] LustreError: 302383:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6244.984835] LustreError: 302383:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6245.719287] Lustre: server umount fs21688-OST0000 complete [ 6250.018737] loop1: detected capacity change from 0 to 204800000 [ 6252.047430] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6252.115484] Lustre: fs21688-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6253.962558] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 6257.005219] loop2: detected capacity change from 0 to 204800000 [ 6257.086629] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6257.090883] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6257.096559] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6258.977049] loop2: detected capacity change from 0 to 204800000 [ 6261.064379] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6261.098865] LustreError: fs21688-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 6261.103851] LustreError: fs21688-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 6261.107865] LustreError: 307342:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 6261.111541] LustreError: 307342:0:(tgt_mount.c:2129:server_put_super()) no obd fs21688-OST0001 [ 6261.113699] LustreError: 307342:0:(tgt_mount.c:116:server_deregister_mount()) fs21688-OST0001 not registered [ 6261.126603] Lustre: server umount fs21688-OST0001 complete [ 6261.127988] LustreError: 307342:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 6261.220263] LustreError: 302381:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6261.225931] LustreError: 302381:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 6264.971703] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6265.033615] Lustre: fs21688-OST0001: new disk, initializing [ 6265.036126] Lustre: srv-fs21688-OST0001: No data found on store. Initialize space. [ 6265.039818] Lustre: 307893:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6265.069581] Lustre: fs21688-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6267.149955] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 6270.818617] Lustre: server umount fs21688-OST0000 complete [ 6274.063167] Lustre: server umount fs21688-OST0001 complete [ 6278.657190] loop1: detected capacity change from 0 to 204800000 [ 6280.611147] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6282.634159] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 6284.889462] loop2: detected capacity change from 0 to 204800000 [ 6286.773926] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6286.843166] Lustre: fs21688-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6286.845106] Lustre: Skipped 1 previous similar message [ 6288.786248] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 6291.875661] Lustre: server umount fs21688-OST0000 complete [ 6295.225864] Lustre: server umount fs21688-OST0001 complete [ 6296.177213] LustreError: 302383:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6296.184234] LustreError: 302383:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 18 previous similar messages [ 6302.177525] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6302.179794] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6302.180717] LustreError: Skipped 1 previous similar message [ 6302.186857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6302.189625] Lustre: Skipped 1 previous similar message [ 6305.495820] Lustre: server umount lustre-OST0000 complete [ 6307.299025] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6307.300108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6307.303336] Lustre: Skipped 2 previous similar messages [ 6312.416602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6312.419166] Lustre: Skipped 2 previous similar messages [ 6317.536346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6317.538327] Lustre: Skipped 1 previous similar message [ 6321.119321] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6321.198182] Lustre: server umount lustre-MDT0000 complete [ 6322.496303] LustreError: 307892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788365286 with bad export cookie 16648696235401977472 [ 6322.499093] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6322.501037] LustreError: 307892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6322.615481] Lustre: server umount lustre-MDT0001 complete [ 6326.028747] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 6327.036480] Key type lgssc unregistered [ 6327.156323] LNet: 313805:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6327.159088] LNetError: 313805:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6327.167292] LNet: Removed LNI 192.168.204.104@tcp [ 6327.466110] Key type .llcrypt unregistered [ 6327.467459] Key type ._llcrypt unregistered [ 6337.585924] Key type ._llcrypt registered [ 6337.587526] Key type .llcrypt registered [ 6337.625684] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_hostid [ 6344.353932] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 6344.700353] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6344.736099] alg: No test for adler32 (adler32-zlib) [ 6345.589695] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 6345.678171] LNet: Added LNI 192.168.204.104@tcp [8/256/0/180] [ 6347.271128] Key type lgssc registered [ 6347.666185] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6359.740521] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6359.747234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6360.823688] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6360.834682] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6360.867130] Lustre: lustre-MDT0000: new disk, initializing [ 6360.887174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6360.893261] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6362.279664] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 6366.599691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6366.650226] Lustre: 317633:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6366.677824] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6366.680157] Lustre: Skipped 1 previous similar message [ 6366.717465] Lustre: lustre-MDT0001: new disk, initializing [ 6366.756458] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6366.772398] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6366.776780] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6368.226241] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 6372.122624] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6374.117119] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6376.434604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6376.540986] Lustre: lustre-OST0000: new disk, initializing [ 6376.542949] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6376.546253] Lustre: 319605:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6376.578784] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6378.647175] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 6382.640244] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6384.314030] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6384.318180] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6384.345807] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6389.750544] Lustre: Mounted lustre-client [ 6390.093833] Lustre: Unmounted lustre-client [ 6394.851823] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6394.852262] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6394.854229] LustreError: Skipped 1 previous similar message [ 6394.854933] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6394.858645] Lustre: Skipped 1 previous similar message [ 6399.257706] Lustre: server umount lustre-OST0000 complete [ 6402.527903] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6402.530844] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6402.536110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6402.537778] Lustre: Skipped 1 previous similar message [ 6405.088724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6405.089682] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6406.834089] Lustre: server umount lustre-MDT0000 complete [ 6408.391323] LustreError: 317628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788365372 with bad export cookie 555916296147517139 [ 6408.394541] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6408.396575] LustreError: 317628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6408.536325] Lustre: server umount lustre-MDT0001 complete [ 6413.183934] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 6414.372838] Key type lgssc unregistered [ 6438.047737] LNet: 321764:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6438.052233] LNetError: 321764:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6513.833557] LNet: Removed LNI 192.168.204.104@tcp [ 6514.414648] Key type .llcrypt unregistered [ 6514.416620] Key type ._llcrypt unregistered [ 6528.414864] Key type ._llcrypt registered [ 6528.418307] Key type .llcrypt registered [ 6528.487920] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_hostid [ 6540.517417] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 6541.370553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6541.396327] alg: No test for adler32 (adler32-zlib) [ 6542.423978] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 6542.603299] LNet: Added LNI 192.168.204.104@tcp [8/256/0/180] [ 6544.248400] Key type lgssc registered [ 6544.968731] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6563.610883] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 12:12:06 (1788365526) [ 6567.063808] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6567.070900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6568.192352] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6568.210547] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6568.261987] Lustre: lustre-MDT0000: new disk, initializing [ 6568.309540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6568.320079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6570.758803] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 6577.149813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6577.216426] Lustre: 324965:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6577.257393] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6577.260566] Lustre: Skipped 1 previous similar message [ 6577.304296] Lustre: lustre-MDT0001: new disk, initializing [ 6577.352574] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6577.372624] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6577.380891] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6579.768501] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 6585.067828] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6588.312567] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6592.268720] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6592.455603] Lustre: lustre-OST0000: new disk, initializing [ 6592.458696] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6592.464104] Lustre: 326936:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6592.532340] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6596.249779] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 6600.233326] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6600.241568] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6600.323877] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6602.331227] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6615.521393] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6615.528359] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6615.538392] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6616.032884] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6616.040308] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6616.048266] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6616.053388] Lustre: Skipped 1 previous similar message [ 6617.573119] Lustre: server umount lustre-OST0000 complete [ 6621.152678] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6621.156931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6621.164847] Lustre: Skipped 1 previous similar message [ 6625.745810] Lustre: server umount lustre-MDT0000 complete [ 6627.876653] LustreError: 324955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788365591 with bad export cookie 17791184209152716713 [ 6627.878526] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6627.884947] LustreError: 324955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6628.059823] Lustre: server umount lustre-MDT0001 complete [ 6635.045463] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 6636.580968] Key type lgssc unregistered [ 6636.807436] LNet: 328838:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6636.812591] LNetError: 328838:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6636.824600] LNet: Removed LNI 192.168.204.104@tcp [ 6637.289168] Key type .llcrypt unregistered [ 6637.290783] Key type ._llcrypt unregistered [ 6648.026828] Key type ._llcrypt registered [ 6648.030620] Key type .llcrypt registered [ 6648.102464] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 6648.513372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6648.523610] alg: No test for adler32 (adler32-zlib) [ 6649.421130] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 6649.536678] LNet: Added LNI 192.168.204.104@tcp [8/256/0/180] [ 6651.135170] Key type lgssc registered [ 6651.692299] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6656.334229] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6656.344267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6657.576332] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6660.392964] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 6666.121339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6666.369972] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6669.195786] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 6673.966960] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6677.161897] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6681.271164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6681.466423] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6684.990442] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 6689.603343] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6699.454226] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6700.357328] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6704.266497] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.4@tcp (stopping) [ 6707.169336] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6707.172319] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6707.182553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6707.190683] Lustre: Skipped 2 previous similar messages [ 6708.735552] Lustre: server umount lustre-MDT0000 complete [ 6709.363227] LustreError: 330090:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6709.371891] LustreError: 330090:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6710.555685] LustreError: 330074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788365674 with bad export cookie 14031109635934264002 [ 6710.560862] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6710.563906] LustreError: 330074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6712.288182] LustreError: 330089:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6712.288426] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6712.304077] Lustre: Skipped 2 previous similar messages [ 6712.306581] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6714.482656] LustreError: 330527:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6714.499054] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.4@tcp (stopping) [ 6716.745741] Lustre: server umount lustre-MDT0001 complete [ 6724.874778] Lustre: server umount lustre-OST0000 complete [ 6730.093132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6732.682419] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 6734.408740] Lustre: 334270:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.104@tcp [ 6735.669435] Lustre: server umount MGS complete [ 6740.676991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6740.960160] LustreError: 334840:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6740.972260] LustreError: 334840:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6741.002902] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6743.344432] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 6746.086956] LustreError: 334841:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6748.808406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6749.021365] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6751.452786] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 6756.796702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6760.697807] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 6765.471421] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6766.525361] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [ 6770.069925] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6772.110069] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 1 sec [ 6774.753554] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6774.759937] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6774.768486] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6774.772557] Lustre: Skipped 1 previous similar message [ 6776.803827] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6776.805171] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6780.551617] Lustre: server umount lustre-OST0000 complete [ 6784.992222] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6784.997132] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6785.005433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6785.015046] Lustre: Skipped 2 previous similar messages [ 6797.279368] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6797.350075] Lustre: server umount lustre-MDT0000 complete [ 6799.128314] LustreError: 336299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788365763 with bad export cookie 14031109635934264842 [ 6799.134101] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6799.134736] LustreError: 336299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6799.278781] Lustre: server umount lustre-MDT0001 complete [ 6804.724284] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 6806.179880] Key type lgssc unregistered [ 6806.364704] LNet: 338275:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6806.367629] LNetError: 338275:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6806.380676] LNet: Removed LNI 192.168.204.104@tcp [ 6806.840550] Key type .llcrypt unregistered [ 6806.842084] Key type ._llcrypt unregistered [ 6817.224599] Key type ._llcrypt registered [ 6817.227121] Key type .llcrypt registered [ 6817.296414] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_hostid [ 6827.963740] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 6828.386457] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6828.402932] alg: No test for adler32 (adler32-zlib) [ 6829.386312] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 6829.594264] LNet: Added LNI 192.168.204.104@tcp [8/256/0/180] [ 6831.271219] Key type lgssc registered [ 6832.034751] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6857.301404] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6857.318237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6858.528676] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6858.577169] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6858.645181] Lustre: lustre-MDT0000: new disk, initializing [ 6858.696938] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6858.712994] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6862.066924] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 6869.919359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6870.014498] Lustre: 341420:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6870.060963] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6870.064485] Lustre: Skipped 1 previous similar message [ 6870.113623] Lustre: lustre-MDT0001: new disk, initializing [ 6870.165759] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6870.188929] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6870.194131] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6873.340125] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 6879.774969] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6883.323668] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6887.563662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6887.757512] Lustre: lustre-OST0000: new disk, initializing [ 6887.765212] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6887.777923] Lustre: 343386:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6887.861332] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6891.601530] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 6894.115729] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6894.127174] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6894.206848] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6897.664734] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6900.067423] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6900.205243] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6902.980813] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6903.107458] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6904.289301] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6904.290892] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6904.300735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6904.307675] Lustre: Skipped 1 previous similar message [ 6909.417354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6909.422878] Lustre: Skipped 1 previous similar message [ 6910.385498] Lustre: server umount lustre-OST0000 complete [ 6914.529921] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6914.537430] Lustre: Skipped 1 previous similar message [ 6914.542539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6914.548127] Lustre: Skipped 1 previous similar message [ 6918.766788] Lustre: server umount lustre-MDT0000 complete [ 6921.046348] LustreError: 341412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788365885 with bad export cookie 18378842652873059117 [ 6921.052046] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6921.058565] LustreError: 341412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6921.236256] Lustre: server umount lustre-MDT0001 complete [ 6926.331653] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 12:18:08 (1788365888) [ 6936.013430] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_hostid [ 6942.639980] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 6963.246578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6963.387833] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6963.409443] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6963.477452] Lustre: lustre-MDT0000: new disk, initializing [ 6963.533841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6963.551479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6966.296279] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 6973.161573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6973.242980] Lustre: 348138:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6973.301970] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6973.304891] Lustre: Skipped 1 previous similar message [ 6973.365279] Lustre: lustre-MDT0001: new disk, initializing [ 6973.462627] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6973.497684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6973.505554] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6976.152979] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 6981.917474] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6984.993722] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6989.050891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6989.217054] Lustre: lustre-OST0000: new disk, initializing [ 6989.220123] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6989.224492] Lustre: 350106:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6989.280302] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6991.176057] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6991.184473] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6991.240503] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6992.638741] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 6998.353285] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7016.930735] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7016.932560] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7016.935839] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7016.939795] Lustre: Skipped 2 previous similar messages [ 7016.946854] Lustre: Skipped 1 previous similar message [ 7022.052652] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7022.057258] Lustre: Skipped 1 previous similar message [ 7022.440219] Lustre: server umount lustre-OST0000 complete [ 7024.608578] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7024.612764] LustreError: Skipped 1 previous similar message [ 7024.615205] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7030.540133] Lustre: server umount lustre-MDT0000 complete [ 7032.289815] LustreError: 348149:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7032.301556] LustreError: 348149:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 7032.735724] LustreError: 350105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788365996 with bad export cookie 18378842652873059999 [ 7032.737021] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7032.742652] LustreError: 350105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7032.927174] Lustre: server umount lustre-MDT0001 complete [ 7038.701734] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 7040.365944] Key type lgssc unregistered [ 7040.549131] LNet: 352256:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7040.559677] LNetError: 352256:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7040.575923] LNet: Removed LNI 192.168.204.104@tcp [ 7041.075892] Key type .llcrypt unregistered [ 7041.077814] Key type ._llcrypt unregistered [ 7086.415903] Key type ._llcrypt registered [ 7086.417380] Key type .llcrypt registered [ 7086.467230] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 7086.601291] Key type .llcrypt unregistered [ 7086.603092] Key type ._llcrypt unregistered [ 7124.467422] Key type ._llcrypt registered [ 7124.468859] Key type .llcrypt registered [ 7124.525782] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 7124.662502] Key type .llcrypt unregistered [ 7124.664498] Key type ._llcrypt unregistered [ 7139.509200] Key type ._llcrypt registered [ 7139.510860] Key type .llcrypt registered [ 7139.565198] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 7139.696577] Key type .llcrypt unregistered [ 7139.698398] Key type ._llcrypt unregistered [ 7154.559896] Key type ._llcrypt registered [ 7154.561119] Key type .llcrypt registered [ 7154.639220] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 7154.809116] Key type .llcrypt unregistered [ 7154.811251] Key type ._llcrypt unregistered [ 7237.445461] Key type ._llcrypt registered [ 7237.449985] Key type .llcrypt registered [ 7237.545619] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 7237.782345] Key type .llcrypt unregistered [ 7237.784612] Key type ._llcrypt unregistered [ 7266.108498] Key type ._llcrypt registered [ 7266.113331] Key type .llcrypt registered [ 7266.213514] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 7266.519950] Key type .llcrypt unregistered [ 7266.522131] Key type ._llcrypt unregistered [ 7289.307225] Key type ._llcrypt registered [ 7289.308621] Key type .llcrypt registered [ 7289.399787] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 7289.664456] Key type .llcrypt unregistered [ 7289.668056] Key type ._llcrypt unregistered [ 7312.171699] Key type ._llcrypt registered [ 7312.173559] Key type .llcrypt registered [ 7312.264953] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 7312.551400] Key type .llcrypt unregistered [ 7312.556277] Key type ._llcrypt unregistered [ 7335.112882] Key type ._llcrypt registered [ 7335.115949] Key type .llcrypt registered [ 7335.228658] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 7335.485442] Key type .llcrypt unregistered [ 7335.490145] Key type ._llcrypt unregistered [ 7359.995507] Key type ._llcrypt registered [ 7359.997243] Key type .llcrypt registered [ 7360.110830] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 7360.361555] Key type .llcrypt unregistered [ 7360.364390] Key type ._llcrypt unregistered [ 7415.502225] Key type ._llcrypt registered [ 7415.508925] Key type .llcrypt registered [ 7415.639301] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 7415.891338] Key type .llcrypt unregistered [ 7415.894470] Key type ._llcrypt unregistered [ 7440.548545] Key type ._llcrypt registered [ 7440.550923] Key type .llcrypt registered [ 7440.636171] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 7440.833468] Key type .llcrypt unregistered [ 7440.837206] Key type ._llcrypt unregistered [ 7468.313312] Key type ._llcrypt registered [ 7468.321053] Key type .llcrypt registered [ 7468.549632] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 7469.091432] Key type .llcrypt unregistered [ 7469.098834] Key type ._llcrypt unregistered [ 7522.138189] Key type ._llcrypt registered [ 7522.146048] Key type .llcrypt registered [ 7522.253707] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_hostid [ 7537.748802] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 7538.677219] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7538.784617] alg: No test for adler32 (adler32-zlib) [ 7539.849286] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 7540.101568] LNet: Added LNI 192.168.204.104@tcp [8/256/0/180] [ 7541.783579] Key type lgssc registered [ 7542.978211] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7575.350123] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7575.365438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7576.576647] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [ 7576.595205] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [ 7576.646767] Lustre: tst250fs-MDT0000: new disk, initializing [ 7576.706508] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7576.722962] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7580.372305] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 7589.008315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7589.096771] Lustre: 372327:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7589.143283] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [ 7589.149942] Lustre: Skipped 1 previous similar message [ 7589.223987] Lustre: tst250fs-MDT0001: new disk, initializing [ 7589.290750] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7589.323022] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7589.331111] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7592.383269] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 7596.714505] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7602.188610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7602.426150] Lustre: tst250fs-OST0000: new disk, initializing [ 7602.430938] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [ 7602.438907] Lustre: 373958:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7602.526142] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7603.072903] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7603.083829] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7603.182945] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7607.236160] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 7616.191813] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7616.315476] Lustre: tst250fs-OST0001: new disk, initializing [ 7616.319656] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [ 7616.324462] Lustre: 374831:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7616.406163] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7620.860282] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 7625.324269] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [ 7626.361224] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7626.374498] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7626.461198] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7627.680770] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [ 7627.683980] Lustre: Skipped 2 previous similar messages [ 7645.664852] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7645.673660] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7645.686086] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 7647.202609] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7647.207541] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 7647.214494] Lustre: Skipped 1 previous similar message [ 7647.218624] Lustre: Skipped 1 previous similar message [ 7649.172981] Lustre: server umount tst250fs-MDT0000 complete [ 7652.159464] LustreError: 372362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788366616 with bad export cookie 14752509455872628323 [ 7652.173043] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7652.179150] LustreError: 372362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7652.328495] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7652.330574] Lustre: tst250fs-MDT0001: Not available for connect from 0@lo (stopping) [ 7652.339614] Lustre: Skipped 2 previous similar messages [ 7652.342644] Lustre: Skipped 2 previous similar messages [ 7652.470571] Lustre: server umount tst250fs-MDT0001 complete [ 7661.673756] Lustre: server umount tst250fs-OST0000 complete [ 7664.809549] Lustre: server umount tst250fs-OST0001 complete [ 7678.835480] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_hostid [ 7686.236977] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 7712.535322] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 12:31:14 (1788366674) [ 7714.252048] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [ 7728.610044] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_hostid [ 7739.537450] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 7770.112437] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7452 sec ========== 12:32:11 (1788366731) [ 7772.014492] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:32:13 (1788366733) === [ 7775.700684] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:32:17 (1788366737) === [ 7809.607685] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 7811.995214] Key type lgssc unregistered [ 7812.237376] LNet: 383972:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7812.242042] LNetError: 383972:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7812.262918] LNet: Removed LNI 192.168.204.104@tcp [ 7813.089191] Key type .llcrypt unregistered [ 7813.096233] Key type ._llcrypt unregistered