[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 529168003 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002401] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004011] kvm-guest: setup PV IPIs [ 0.007485] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.011087] LSM: Security Framework initializing [ 0.012055] Yama: becoming mindful. [ 0.014041] SELinux: Initializing. [ 0.015074] *** VALIDATE selinux *** [ 0.023321] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027790] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028167] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030129] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031119] *** VALIDATE tmpfs *** [ 0.033378] *** VALIDATE proc *** [ 0.034228] *** VALIDATE cgroup *** [ 0.035010] *** VALIDATE cgroup2 *** [ 0.036291] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038145] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040030] Spectre V2 : User space: Vulnerable [ 0.041009] Speculative Store Bypass: Vulnerable [ 0.043715] debug: unmapping init [mem 0xffffffffb9e59000-0xffffffffb9e60fff] [ 0.045152] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046698] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047022] ... version: 2 [ 0.048011] ... bit width: 48 [ 0.049010] ... generic registers: 4 [ 0.050011] ... value mask: 0000ffffffffffff [ 0.051013] ... max period: 00007fffffffffff [ 0.052014] ... fixed-purpose events: 3 [ 0.053012] ... event mask: 000000070000000f [ 0.054329] rcu: Hierarchical SRCU implementation. [ 0.056518] smp: Bringing up secondary CPUs ... [ 0.057670] x86: Booting SMP configuration: [ 0.058022] .... node #0, CPUs: #1 #2 #3 [ 0.061392] smp: Brought up 1 node, 4 CPUs [ 0.063012] smpboot: Max logical packages: 1 [ 0.064017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.137034] node 0 deferred pages initialised in 70ms [ 0.141138] devtmpfs: initialized [ 0.142292] x86/mm: Memory block size: 128MB [ 0.144000] gcov: version magic: 0x41383552 [ 0.147420] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.151118] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.153336] pinctrl core: initialized pinctrl subsystem [ 0.156186] [ 0.156835] ************************************************************* [ 0.159013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.161010] ** ** [ 0.163021] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.166016] ** ** [ 0.168016] ** This means that this kernel is built to expose internal ** [ 0.171017] ** IOMMU data structures, which may compromise security on ** [ 0.173013] ** your system. ** [ 0.176018] ** ** [ 0.178014] ** If you see this message and you are not debugging the ** [ 0.180016] ** kernel, report this immediately to your vendor! ** [ 0.182014] ** ** [ 0.184013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.186014] ************************************************************* [ 0.189089] NET: Registered protocol family 16 [ 0.191474] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.194073] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.197071] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.201037] cpuidle: using governor menu [ 0.202118] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.204682] PCI: Using configuration type 1 for base access [ 0.205000] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.214165] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.217056] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.220171] cryptd: max_cpu_qlen set to 1000 [ 0.223339] ACPI: Added _OSI(Module Device) [ 0.225018] ACPI: Added _OSI(Processor Device) [ 0.227018] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.229016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.234896] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.242483] ACPI: Interpreter enabled [ 0.244086] ACPI: PM: (supports S0 S3 S4 S5) [ 0.245015] ACPI: Using IOAPIC for interrupt routing [ 0.247139] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.252116] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.266470] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.269045] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.272021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.277083] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.282629] acpiphp: Slot [2] registered [ 0.284200] acpiphp: Slot [5] registered [ 0.286180] acpiphp: Slot [6] registered [ 0.287185] acpiphp: Slot [7] registered [ 0.289152] acpiphp: Slot [8] registered [ 0.291214] acpiphp: Slot [9] registered [ 0.292152] acpiphp: Slot [10] registered [ 0.294180] acpiphp: Slot [3] registered [ 0.296116] acpiphp: Slot [4] registered [ 0.298176] acpiphp: Slot [11] registered [ 0.300112] acpiphp: Slot [12] registered [ 0.301108] acpiphp: Slot [13] registered [ 0.304097] acpiphp: Slot [14] registered [ 0.306138] acpiphp: Slot [15] registered [ 0.308132] acpiphp: Slot [16] registered [ 0.310199] acpiphp: Slot [17] registered [ 0.312139] acpiphp: Slot [18] registered [ 0.315135] acpiphp: Slot [19] registered [ 0.316192] acpiphp: Slot [20] registered [ 0.318090] acpiphp: Slot [21] registered [ 0.319100] acpiphp: Slot [22] registered [ 0.320147] acpiphp: Slot [23] registered [ 0.322120] acpiphp: Slot [24] registered [ 0.324123] acpiphp: Slot [25] registered [ 0.326155] acpiphp: Slot [26] registered [ 0.328203] acpiphp: Slot [27] registered [ 0.330135] acpiphp: Slot [28] registered [ 0.332136] acpiphp: Slot [29] registered [ 0.334246] acpiphp: Slot [30] registered [ 0.337163] acpiphp: Slot [31] registered [ 0.338111] PCI host bridge to bus 0000:00 [ 0.340023] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.343036] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.347045] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.350031] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.352034] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.355035] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.357234] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.361233] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.366098] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.380018] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.385067] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.388023] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.392022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.394016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.397786] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.401124] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.405074] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.411060] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.418017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.432024] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.439017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.445000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.455069] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.463016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.476000] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.483835] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.491020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.496017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.511016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.522618] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.527015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.535015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.559017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.569000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.578016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.584017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.599019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.608639] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.625013] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.636017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.665019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.686861] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.697023] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.706023] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.726021] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.742520] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.744371] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.747380] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.751722] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.755270] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.761117] iommu: Default domain type: Passthrough [ 0.764684] SCSI subsystem initialized [ 0.767253] ACPI: bus type USB registered [ 0.770185] usbcore: registered new interface driver usbfs [ 0.773128] usbcore: registered new interface driver hub [ 0.776125] usbcore: registered new device driver usb [ 0.778279] pps_core: LinuxPPS API ver. 1 registered [ 0.781019] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.786071] PTP clock support registered [ 0.789049] EDAC MC: Ver: 3.0.0 [ 0.791714] PCI: Using ACPI for IRQ routing [ 0.794439] NetLabel: Initializing [ 0.797018] NetLabel: domain hash size = 128 [ 0.799011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.801088] NetLabel: unlabeled traffic allowed by default [ 0.804564] vgaarb: loaded [ 0.806392] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.808014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.815732] clocksource: Switched to clocksource kvm-clock [ 0.934691] VFS: Disk quotas dquot_6.6.0 [ 0.937059] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.940143] *** VALIDATE ramfs *** [ 0.941991] *** VALIDATE hugetlbfs *** [ 0.944378] pnp: PnP ACPI init [ 0.947532] pnp: PnP ACPI: found 6 devices [ 0.975053] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.981936] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.984517] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.988833] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.991597] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.994787] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.998383] NET: Registered protocol family 2 [ 1.001035] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.006095] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.009996] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.015885] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.019858] TCP: Hash tables configured (established 65536 bind 65536) [ 1.023051] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.026522] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.029795] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.033786] NET: Registered protocol family 1 [ 1.037251] RPC: Registered named UNIX socket transport module. [ 1.041229] RPC: Registered udp transport module. [ 1.043818] RPC: Registered tcp transport module. [ 1.046623] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.050587] NET: Registered protocol family 44 [ 1.053485] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.056823] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.060432] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.064249] PCI: CLS 0 bytes, default 64 [ 1.066758] Unpacking initramfs... [ 2.519260] debug: unmapping init [mem 0xffff95a8fcc54000-0xffff95a8fffbffff] [ 2.523714] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.526392] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.529207] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.086349] Initialise system trusted keyrings [ 3.088202] Key type blacklist registered [ 3.090870] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.100966] zbud: loaded [ 3.104779] *** VALIDATE nfs *** [ 3.106473] *** VALIDATE nfs4 *** [ 3.108345] pstore: using deflate compression [ 3.113419] Platform Keyring initialized [ 3.261890] NET: Registered protocol family 38 [ 3.264060] Key type asymmetric registered [ 3.265727] Asymmetric key parser 'x509' registered [ 3.268133] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.272342] io scheduler mq-deadline registered [ 3.274305] io scheduler kyber registered [ 3.276407] io scheduler bfq registered [ 3.279177] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.283406] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.287753] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.291353] ACPI: Power Button [PWRF] [ 3.297115] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.305214] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.328258] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.339958] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.363304] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.394257] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.428420] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.433767] Non-volatile memory driver v1.3 [ 3.435575] Linux agpgart interface v0.103 [ 3.476985] virtio_blk virtio1: [vda] 146704 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.482194] vda: detected capacity change from 0 to 75112448 [ 3.508771] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.512755] vdb: detected capacity change from 0 to 1073741824 [ 3.528761] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.531634] vdc: detected capacity change from 0 to 2621440000 [ 3.548048] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.551135] vdd: detected capacity change from 0 to 2621440000 [ 3.570725] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.574122] vde: detected capacity change from 0 to 4294967296 [ 3.597549] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.600901] vdf: detected capacity change from 0 to 4294967296 [ 3.628476] libphy: Fixed MDIO Bus: probed [ 3.640075] usbcore: registered new interface driver usbserial_generic [ 3.641848] usbserial: USB Serial support registered for generic [ 3.643683] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.651866] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.655941] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.658988] mousedev: PS/2 mouse device common for all mice [ 3.661894] rtc_cmos 00:05: RTC can wake from S4 [ 3.664299] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.667851] rtc_cmos 00:05: registered as rtc0 [ 3.670260] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.671426] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.673868] intel_pstate: CPU model not supported [ 3.678835] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.683867] hid: raw HID events driver (C) Jiri Kosina [ 3.686790] usbcore: registered new interface driver usbhid [ 3.689325] usbhid: USB HID core driver [ 3.691222] drop_monitor: Initializing network drop monitor service [ 3.694238] Initializing XFRM netlink socket [ 3.696489] NET: Registered protocol family 10 [ 3.701660] Segment Routing with IPv6 [ 3.703486] NET: Registered protocol family 17 [ 3.706099] mpls_gso: MPLS GSO support [ 3.712992] RAS: Correctable Errors collector initialized. [ 3.715445] AVX version of gcm_enc/dec engaged. [ 3.718018] AES CTR mode by8 optimization enabled [ 3.809450] sched_clock: Marking stable (3809420070, 0)->(4743287785, -933867715) [ 3.813595] registered taskstats version 1 [ 3.817108] Loading compiled-in X.509 certificates [ 3.820403] zswap: loaded using pool lzo/zbud [ 3.850494] Key type big_key registered [ 3.864299] Key type encrypted registered [ 3.866319] ima: No TPM chip found, activating TPM-bypass! [ 3.868766] ima: Allocated hash algorithm: sha1 [ 3.870941] ima: No architecture policies found [ 3.872928] evm: Initialising EVM extended attributes: [ 3.875556] evm: security.selinux [ 3.876764] evm: security.ima [ 3.877906] evm: security.capability [ 3.900012] evm: HMAC attrs: 0x1 [ 3.910504] rtc_cmos 00:05: setting system clock to 2026-09-19 05:15:06 UTC (1789794906) [ 3.917712] debug: unmapping init [mem 0xffffffffbae03000-0xffffffffbaffffff] [ 3.922376] debug: unmapping init [mem 0xffffffffb9b82000-0xffffffffb9e58fff] [ 3.931117] Write protecting the kernel read-only data: 28672k [ 3.935505] debug: unmapping init [mem 0xffffffffb8203000-0xffffffffb83fffff] [ 3.938864] debug: unmapping init [mem 0xffffffffb8b14000-0xffffffffb8bfffff] [ 3.983162] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.993930] systemd[1]: Detected virtualization kvm. [ 3.996131] systemd[1]: Detected architecture x86-64. [ 3.998310] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.027260] systemd[1]: No hostname configured. [ 4.029163] systemd[1]: Set hostname to . [ 4.031443] random: systemd: uninitialized urandom read (16 bytes read) [ 4.034308] systemd[1]: Initializing machine ID from random generator. [ 4.108659] random: ln: uninitialized urandom read (6 bytes read) [ 4.225190] random: systemd: uninitialized urandom read (16 bytes read) [ 4.228549] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 4.235503] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.241895] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local Encrypted Volumes. Starting Setup Virtual Console... [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.013324] device-mapper: uevent: version 1.0.3 [ 5.016499] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [[ 5.491237] random: fast init done  OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ 5.945885] virtio_net virtio0 ens2: renamed from eth0 [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 6.048499] scsi host0: ata_piix [ 6.091758] scsi host1: ata_piix [ 6.096342] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 6.098950] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.035279] dracut-initqueue[586]: RTNETLINK answers: File exists [ 10.414321] random: crng init done [ 10.415677] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 11.463624] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.936376] printk: systemd: 26 output lines suppressed due to ratelimiting [ 13.242395] SELinux: Disabled at runtime. [ 13.315119] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 13.327147] systemd[1]: Detected virtualization kvm. [ 13.329770] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.930498] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.937288] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.948525] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.952597] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.958909] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.974881] systemd[1]: Starting Journal Service... Starting Journal Service... [ 14.005898] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. Starting Remount Root and Kernel File Systems... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-getty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Control Socket. [ 14.093328] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Mounting Kernel Debug File System... Starting Apply Kernel Variables... Starting udev Coldplug all Devices... Mounting Huge Pages File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 14.626253] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 15.067158] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 15.135696] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 15.269223] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 15.290611] EDAC sbridge: Ver: 1.1.2 [ 17.210482] Key type dns_resolver registered [ 17.544270] NFS: Registering the id_resolver key type [ 17.548150] Key type id_resolver registered [ 17.549903] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg420-server login: [ 52.825160] libcfs: loading out-of-tree module taints kernel. [ 52.858074] Key type ._llcrypt registered [ 52.859897] Key type .llcrypt registered [ 52.937889] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 69.033050] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 70.177288] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 70.190961] alg: No test for adler32 (adler32-zlib) [ 71.485617] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [ 72.237460] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 73.952201] Key type lgssc registered [ 75.118901] Lustre: Echo OBD driver; http://www.lustre.org/ [ 88.489372] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 125.960532] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 136.794933] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 136.828290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 138.096188] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 138.133335] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 138.232535] Lustre: lustre-MDT0000: new disk, initializing [ 138.299369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 138.316522] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 142.106342] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 154.183187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 154.289338] Lustre: 6510:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 154.321223] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 154.326352] Lustre: Skipped 1 previous similar message [ 154.394566] Lustre: lustre-MDT0001: new disk, initializing [ 154.454656] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 154.488660] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 154.497913] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 158.460241] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 162.997564] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 172.152751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 172.366654] Lustre: lustre-OST0000: new disk, initializing [ 172.370780] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 172.382690] Lustre: 8445:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 172.449652] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 173.396731] hrtimer: interrupt took 1804122 ns [ 177.711544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 177.718630] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 177.783382] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 177.809705] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 190.689775] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 190.843460] Lustre: lustre-OST0001: new disk, initializing [ 190.850278] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 190.861267] Lustre: 9517:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 190.914561] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 196.657819] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 199.200327] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 199.216798] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 199.276242] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 208.084812] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 213.501629] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 219.933510] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing check_logdir /tmp/testlogs/ [ 224.057400] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing yml_node [ 227.351683] Lustre: DEBUG MARKER: Client: 2.17.57.110 [ 229.338649] Lustre: DEBUG MARKER: MDS: 2.17.57.110 [ 231.240949] Lustre: DEBUG MARKER: OSS: 2.17.57.110 [ 232.632773] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Sep 19 01:18:53 EDT 2026 [ 246.676956] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 248.125217] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 265.703285] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 265.705438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 265.721263] Lustre: Skipped 2 previous similar messages [ 265.725991] Lustre: Skipped 1 previous similar message [ 269.198118] Lustre: server umount lustre-MDT0000 complete [ 272.098679] LustreError: 6501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789795174 with bad export cookie 9110633904387219517 [ 272.099968] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 272.106432] LustreError: 6501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 275.936786] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 275.937500] LustreError: 9816:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 275.951399] Lustre: Skipped 1 previous similar message [ 275.952203] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 275.952207] Lustre: Skipped 2 previous similar messages [ 275.978651] LustreError: 9816:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 277.985688] LustreError: 6517:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 277.987316] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 278.008542] LustreError: 6517:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 278.016749] Lustre: Skipped 2 previous similar messages [ 278.405407] Lustre: server umount lustre-MDT0001 complete [ 281.611593] Lustre: server umount lustre-OST0000 complete [ 284.529042] Lustre: server umount lustre-OST0001 complete [ 289.677375] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 296.719104] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 321.314441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 321.511257] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 321.543530] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 321.601451] Lustre: lustre-MDT0000: new disk, initializing [ 321.678590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 321.692483] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 325.229111] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 332.779794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 332.875565] Lustre: 15066:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 332.933978] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 332.937639] Lustre: Skipped 1 previous similar message [ 332.997922] Lustre: lustre-MDT0001: new disk, initializing [ 333.087147] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 333.118392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 333.132476] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 336.287988] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 343.303985] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 347.891887] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 352.831530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 353.038421] Lustre: lustre-OST0000: new disk, initializing [ 353.041711] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 353.051052] Lustre: 17035:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 353.140651] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 355.000312] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 355.006203] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 355.118426] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 357.224702] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 363.777369] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 367.083681] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 367.278747] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 370.932820] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 371.089660] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 375.785450] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 375.790803] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 375.793808] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 375.797521] Lustre: Skipped 2 previous similar messages [ 375.817100] LustreError: Skipped 1 previous similar message [ 378.770170] Lustre: server umount lustre-OST0000 complete [ 384.489367] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 384.505842] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 384.523057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 384.531618] Lustre: Skipped 1 previous similar message [ 387.607152] Lustre: server umount lustre-MDT0000 complete [ 390.292702] LustreError: 15058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789795292 with bad export cookie 9110633904387221050 [ 390.293982] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 390.307044] LustreError: 15058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 390.561541] Lustre: server umount lustre-MDT0001 complete [ 395.167903] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 01:21:35 (1789795295) [ 410.796269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 411.054209] LustreError: 19908:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 411.118143] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 414.718957] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 416.226185] LustreError: 19909:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 421.089259] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 424.854633] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 431.017451] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 435.578362] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 437.571978] Lustre: Failing over lustre-MDT0000 [ 437.865774] Lustre: server umount lustre-MDT0000 complete [ 440.803235] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 440.807848] LustreError: 19905:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 440.825737] Lustre: Skipped 2 previous similar messages [ 452.065721] LustreError: 20635:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 452.086109] LustreError: 20635:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 456.444249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 456.569719] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 460.909509] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 462.323224] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 462.329852] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 462.353994] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 469.579195] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 476.645598] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 476.647360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 476.660894] Lustre: Skipped 1 previous similar message [ 476.671717] Lustre: Skipped 3 previous similar messages [ 487.392208] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 487.546250] Lustre: server umount lustre-MDT0000 complete [ 490.531323] LustreError: 19889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789795393 with bad export cookie 9110633904387222464 [ 490.538168] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 490.548463] LustreError: 19889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 490.799280] Lustre: server umount lustre-MDT0001 complete [ 497.508243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 497.888352] LustreError: 23241:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 497.952934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 497.961810] Lustre: Skipped 2 previous similar messages [ 501.554621] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 508.361658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 512.283928] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 517.749837] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 521.944953] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 523.582742] Lustre: Failing over lustre-MDT0000 [ 523.699102] Lustre: server umount lustre-MDT0000 complete [ 524.259824] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 524.264303] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 533.473391] LustreError: 24336:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 533.487799] LustreError: 24336:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 540.452488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 540.557644] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 544.271938] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 546.273330] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 546.279450] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 546.289137] Lustre: Skipped 1 previous similar message [ 546.303703] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 552.291756] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 556.514671] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 556.523977] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 556.538433] Lustre: Skipped 1 previous similar message [ 556.547503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 556.559171] Lustre: Skipped 4 previous similar messages [ 561.554589] Lustre: server umount lustre-MDT0000 complete [ 564.469085] LustreError: 23221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789795467 with bad export cookie 9110633904387223402 [ 564.474704] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 564.479974] LustreError: 23221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 564.713659] Lustre: server umount lustre-MDT0001 complete [ 570.854989] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 01:24:31 (1789795471) [ 583.348101] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 590.578334] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 621.475740] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 630.353135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 630.514552] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 630.541785] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 630.669394] Lustre: lustre-MDT0000: new disk, initializing [ 630.745434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 630.749672] Lustre: Skipped 2 previous similar messages [ 630.767117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 634.549727] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 643.981581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 644.104902] Lustre: 29610:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 644.321785] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 648.320327] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 656.868833] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 662.008696] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 668.433938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 668.746778] Lustre: 31578:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 670.679348] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 670.761730] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 670.876289] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 670.893068] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 674.554785] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 683.230789] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 691.553761] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 710.522825] Lustre: server umount lustre-OST0000 complete [ 716.258089] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 716.268281] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 716.281033] Lustre: Skipped 1 previous similar message [ 716.285570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 716.291525] Lustre: Skipped 2 previous similar messages [ 724.066057] Lustre: server umount lustre-MDT0000 complete [ 726.999074] LustreError: 29604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789795629 with bad export cookie 9110633904387223801 [ 727.010743] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 727.017110] LustreError: 29604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 737.408555] Lustre: server umount lustre-MDT0001 complete [ 745.691239] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 747.989045] Key type lgssc unregistered [ 748.231431] LNet: 33555:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 748.254369] LNetError: 33555:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 748.273559] LNet: Removed LNI 192.168.204.120@tcp [ 748.989158] Key type .llcrypt unregistered [ 748.990870] Key type ._llcrypt unregistered [ 769.220187] Key type ._llcrypt registered [ 769.222740] Key type .llcrypt registered [ 769.318517] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 769.969217] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 770.017151] alg: No test for adler32 (adler32-zlib) [ 771.017744] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [ 771.247929] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 772.904166] Key type lgssc registered [ 773.764901] Lustre: Echo OBD driver; http://www.lustre.org/ [ 779.610765] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 779.629663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 780.933050] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 784.092723] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 790.309265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 790.503756] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 790.605881] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 794.163589] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 799.666768] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 803.811940] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 808.699455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 808.937478] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 813.101330] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 816.125241] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 818.388876] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 840.750890] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 01:29:01 (1789795741) [ 842.260452] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 843.713830] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 848.874993] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 01:29:09 (1789795749) [ 855.709834] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 01:29:16 (1789795756) [ 856.586380] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 862.581767] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 862.588801] Lustre: Skipped 2 previous similar messages [ 863.832661] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 01:29:24 (1789795764) [ 866.801186] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 866.803845] Lustre: Skipped 3 previous similar messages [ 879.235197] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 879.245696] Lustre: Skipped 6 previous similar messages [ 881.170889] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 01:29:42 (1789795782) [ 883.215671] Lustre: *** cfs_fail_loc=131b, val=2*** [ 885.725763] Lustre: *** cfs_fail_loc=131b, val=2*** [ 891.820259] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 01:29:52 (1789795792) [ 896.292198] Lustre: Disabling parameter general.jobid_name=TESTNAME1 in log params [ 896.296090] Lustre: Skipped 6 previous similar messages [ 902.385070] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 01:30:03 (1789795803) [ 913.379089] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 913.384153] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 913.388352] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 913.391500] Lustre: Skipped 1 previous similar message [ 913.402021] Lustre: Skipped 1 previous similar message [ 918.499957] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 918.503484] Lustre: Skipped 1 previous similar message [ 923.622362] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 923.628088] Lustre: Skipped 1 previous similar message [ 925.152153] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 925.245863] Lustre: server umount lustre-OST0000 complete [ 928.739145] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 928.740848] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 928.743712] LustreError: Skipped 1 previous similar message [ 928.752922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 933.626538] Lustre: server umount lustre-MDT0000 complete [ 933.857661] LustreError: 34962:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 933.866508] LustreError: 34962:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 935.853471] LustreError: 38900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789795838 with bad export cookie 2163567321297511200 [ 935.854626] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 935.860581] LustreError: 38900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 936.056760] Lustre: server umount lustre-MDT0001 complete [ 943.290813] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 945.182706] Key type lgssc unregistered [ 945.413802] LNet: 42837:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 945.425449] LNetError: 42837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 945.444783] LNet: Removed LNI 192.168.204.120@tcp [ 946.012752] Key type .llcrypt unregistered [ 946.014677] Key type ._llcrypt unregistered [ 958.906870] Key type ._llcrypt registered [ 958.908842] Key type .llcrypt registered [ 958.993577] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 970.438395] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 970.939149] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 970.961383] alg: No test for adler32 (adler32-zlib) [ 971.892286] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [ 972.038218] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 973.657776] Key type lgssc registered [ 974.452645] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1000.727467] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1000.744508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1001.930586] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1001.972509] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1002.028675] Lustre: lustre-MDT0000: new disk, initializing [ 1002.097715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1002.113519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1005.676811] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1013.539301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1013.619360] Lustre: 45963:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1013.674857] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1013.680686] Lustre: Skipped 1 previous similar message [ 1013.771882] Lustre: lustre-MDT0001: new disk, initializing [ 1013.854151] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1013.883820] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1013.894958] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1016.937713] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1023.942625] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1028.074280] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1032.755484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1032.978300] Lustre: lustre-OST0000: new disk, initializing [ 1032.987752] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1033.000218] Lustre: 47930:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1033.081333] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1037.409191] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1039.903587] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1039.909665] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1040.015789] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1045.481072] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1049.020256] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1049.183560] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1052.620552] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1052.757905] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1055.206841] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1055.214754] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1055.231536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1055.715566] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1055.737783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1060.251926] Lustre: server umount lustre-OST0000 complete [ 1064.931034] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1064.936191] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1064.945926] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1068.935211] Lustre: server umount lustre-MDT0000 complete [ 1071.171705] LustreError: 45957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789795973 with bad export cookie 8238254610816348103 [ 1071.173766] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1071.179399] LustreError: 45957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1071.397995] Lustre: server umount lustre-MDT0001 complete [ 1078.095720] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 01:32:59 (1789795979) [ 1082.526537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1082.814747] LustreError: 50387:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1082.827823] LustreError: 50387:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1082.872086] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1085.617710] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1087.970806] LustreError: 50388:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1090.666612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1090.880348] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1093.545481] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1098.131457] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1101.627473] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1105.703817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1105.930292] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1109.655986] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1114.051771] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1121.482891] LustreError: 52915:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1122.188455] Lustre: Modifying parameter general.timeout=1 in log params [ 1123.437878] Lustre: Modifying parameter general.timeout=3 in log params [ 1123.441701] Lustre: Skipped 1 previous similar message [ 1125.869969] Lustre: Modifying parameter general.timeout=7 in log params [ 1125.872382] Lustre: Skipped 3 previous similar messages [ 1130.167150] Lustre: Modifying parameter general.timeout=14 in log params [ 1130.174771] Lustre: Skipped 6 previous similar messages [ 1138.824702] Lustre: Modifying parameter general.timeout=28 in log params [ 1138.829544] Lustre: Skipped 13 previous similar messages [ 1155.001371] Lustre: Modifying parameter general.timeout=55 in log params [ 1155.005640] Lustre: Skipped 26 previous similar messages [ 1187.162159] Lustre: Modifying parameter general.timeout=114 in log params [ 1187.165325] Lustre: Skipped 58 previous similar messages [ 1202.686877] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 01:35:03 (1789796103) [ 1204.595748] Lustre: 59831:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1205.712847] Lustre: 59929:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1208.267561] Lustre: 60142:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1208.276879] Lustre: 60142:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1224.733119] Lustre: 61682:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1224.744903] Lustre: 61682:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1224.767223] LustreError: 61682:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1227.286906] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 01:35:28 (1789796128) [ 1237.476217] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1237.483599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1237.493647] Lustre: Skipped 3 previous similar messages [ 1237.502150] Lustre: Skipped 4 previous similar messages [ 1240.773360] Lustre: server umount lustre-MDT0000 complete [ 1242.595679] LustreError: 51308:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1242.606632] LustreError: 51308:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1243.305240] LustreError: 54743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789796145 with bad export cookie 8238254610816348985 [ 1243.311110] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1243.312903] LustreError: 54743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1243.516317] Lustre: server umount lustre-MDT0001 complete [ 1256.511989] Lustre: server umount lustre-OST0000 complete [ 1273.101189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1273.237874] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1273.256353] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1273.261110] Lustre: Skipped 38 previous similar messages [ 1273.372752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1275.583129] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1280.265304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1280.331941] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1280.349486] Lustre: 64208:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1283.004980] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1287.403267] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1290.646906] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1294.682862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1294.799869] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1298.386066] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1302.860841] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1304.064526] Lustre: 66774:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1310.001425] Lustre: server umount lustre-OST0000 complete [ 1316.322624] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1316.328619] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1316.336869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1322.516426] Lustre: server umount lustre-MDT0000 complete [ 1324.950713] LustreError: 66067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789796227 with bad export cookie 8238254610816351631 [ 1324.951937] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1324.958714] LustreError: 66067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1335.361383] Lustre: server umount lustre-MDT0001 complete [ 1342.503917] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 1344.356634] Key type lgssc unregistered [ 1344.549711] LNet: 67978:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1344.557561] LNetError: 67978:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1344.569821] LNet: Removed LNI 192.168.204.120@tcp [ 1345.048788] Key type .llcrypt unregistered [ 1345.051269] Key type ._llcrypt unregistered [ 1361.058206] Key type ._llcrypt registered [ 1361.059700] Key type .llcrypt registered [ 1361.122744] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 1361.809815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1361.821321] alg: No test for adler32 (adler32-zlib) [ 1362.765119] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [ 1362.949854] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 1364.596087] Key type lgssc registered [ 1365.386534] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1370.741386] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1370.763324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1372.115704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1375.129391] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1381.112659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1381.552772] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1384.581549] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1389.713989] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1393.654715] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1398.289597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1398.668261] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1402.744871] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1407.073853] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1427.210202] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1432.545995] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1432.550483] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1432.558585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1433.569847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1433.570926] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1438.292606] Lustre: server umount lustre-MDT0000 complete [ 1438.689790] LustreError: 71212:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1438.697436] LustreError: 71212:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1440.459292] LustreError: 69345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789796343 with bad export cookie 2984362870528554297 [ 1440.467860] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1440.468832] LustreError: 69345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1440.666200] Lustre: server umount lustre-MDT0001 complete [ 1453.274815] Lustre: server umount lustre-OST0000 complete [ 1469.863543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1470.031215] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1470.049586] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1470.210510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1472.778790] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1478.088494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1478.200639] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1478.238251] Lustre: 74256:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1478.625150] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1481.103763] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1485.322450] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1488.397287] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1492.299280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1492.397971] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1492.641283] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1496.370428] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1500.381655] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1506.298688] Lustre: 76833:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1507.175530] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1512.418442] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1512.421880] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1512.427525] Lustre: Skipped 1 previous similar message [ 1512.430252] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1512.432608] Lustre: Skipped 3 previous similar messages [ 1515.452766] Lustre: server umount lustre-OST0000 complete [ 1517.539139] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1517.541462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1517.553168] Lustre: Skipped 2 previous similar messages [ 1522.658106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1522.661443] Lustre: Skipped 2 previous similar messages [ 1523.704171] Lustre: server umount lustre-MDT0000 complete [ 1525.870766] LustreError: 74248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789796428 with bad export cookie 2984362870528555116 [ 1525.872352] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1525.877632] LustreError: 74248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1526.041300] Lustre: server umount lustre-MDT0001 complete [ 1531.984495] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 1533.457184] Key type lgssc unregistered [ 1533.634688] LNet: 78040:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1533.642628] LNetError: 78040:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1533.662711] LNet: Removed LNI 192.168.204.120@tcp [ 1534.095579] Key type .llcrypt unregistered [ 1534.096701] Key type ._llcrypt unregistered [ 1548.062995] Key type ._llcrypt registered [ 1548.065379] Key type .llcrypt registered [ 1548.137122] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 1548.762777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1548.781582] alg: No test for adler32 (adler32-zlib) [ 1549.718971] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [ 1549.838042] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 1551.440618] Key type lgssc registered [ 1552.047992] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1556.035789] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1556.048387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1557.278379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1559.453937] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1564.332660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1564.653385] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1566.637545] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1570.200289] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1572.822947] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1576.418550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1576.753832] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1579.846921] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1583.401784] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1591.701882] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1592.632109] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1592.634711] Lustre: Skipped 1 previous similar message [ 1593.877562] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1593.885339] Lustre: Skipped 2 previous similar messages [ 1596.083068] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1596.088992] Lustre: Skipped 4 previous similar messages [ 1600.104307] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1600.107971] Lustre: Skipped 8 previous similar messages [ 1608.350662] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1608.356254] Lustre: Skipped 17 previous similar messages [ 1624.568687] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1624.574070] Lustre: Skipped 34 previous similar messages [ 1640.675445] Lustre: 87501:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1641.818890] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 01:42:23 (1789796543) [ 1642.529200] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1643.369690] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 01:42:24 (1789796544) [ 1646.565791] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1646.566227] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1646.571088] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1646.576730] Lustre: Skipped 1 previous similar message [ 1651.689240] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1651.696251] Lustre: Skipped 1 previous similar message [ 1656.804151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1656.809098] Lustre: Skipped 1 previous similar message [ 1659.360564] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1659.435719] Lustre: server umount lustre-OST0000 complete [ 1661.922698] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1661.924335] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1661.924898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1661.932123] Lustre: Skipped 2 previous similar messages [ 1667.041339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1667.045632] Lustre: Skipped 2 previous similar messages [ 1667.497147] Lustre: server umount lustre-MDT0000 complete [ 1669.441492] LustreError: 79408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789796572 with bad export cookie 11461322809421851063 [ 1669.445065] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1669.447113] LustreError: 79408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1669.650796] Lustre: server umount lustre-MDT0001 complete [ 1676.448413] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 1677.924081] Key type lgssc unregistered [ 1678.105753] LNet: 88747:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1678.109473] LNetError: 88747:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1678.123817] LNet: Removed LNI 192.168.204.120@tcp [ 1678.628680] Key type .llcrypt unregistered [ 1678.630707] Key type ._llcrypt unregistered [ 1680.601640] Key type ._llcrypt registered [ 1680.603657] Key type .llcrypt registered [ 1680.662707] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1682.676987] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules [ 1683.150535] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1683.233846] alg: No test for adler32 (adler32-zlib) [ 1684.136218] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [ 1684.250692] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 1685.864657] Key type lgssc registered [ 1686.407230] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1686.938791] LustreError: 89894:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 1696.560789] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 1700.828432] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1708.065727] LustreError: 89894:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1708.082368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1709.369241] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1711.295635] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1715.143184] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 01:43:36 (1789796616) [ 1717.174136] LustreError: 90950:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1717.179135] LustreError: 90950:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1717.182906] LustreError: 90950:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 0, failed: rc = -5 [ 1717.332616] Lustre: server umount lustre-MDT0000 complete [ 1723.110133] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 1724.530607] Key type lgssc unregistered [ 1724.683731] LNet: 92441:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1724.687618] LNetError: 92441:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1724.697912] LNet: Removed LNI 192.168.204.120@tcp [ 1725.157934] Key type .llcrypt unregistered [ 1725.160855] Key type ._llcrypt unregistered [ 1734.011666] Key type ._llcrypt registered [ 1734.013226] Key type .llcrypt registered [ 1734.054823] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 1734.600954] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1734.616677] alg: No test for adler32 (adler32-zlib) [ 1735.508829] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [ 1735.645524] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 1737.248169] Key type lgssc registered [ 1737.686689] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1741.207507] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1741.215862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1742.481723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1744.436966] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1748.453182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1748.829526] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1750.978912] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1754.655109] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1757.298477] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1760.695305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1761.017629] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1763.883855] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1767.592606] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1804.404176] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 01:45:05 (1789796705) [ 1805.204434] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 1806.047953] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 01:45:07 (1789796707) [ 1810.403821] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1810.405755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1810.414619] Lustre: Skipped 1 previous similar message [ 1810.414904] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1810.421668] Lustre: Skipped 1 previous similar message [ 1814.072781] Lustre: server umount lustre-MDT0000 complete [ 1815.521324] LustreError: 94786:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1815.530097] LustreError: 94786:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1815.870807] LustreError: 93613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789796718 with bad export cookie 14578166137505944106 [ 1815.876338] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1815.878225] LustreError: 93613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1816.012608] Lustre: server umount lustre-MDT0001 complete [ 1828.188294] Lustre: server umount lustre-OST0000 complete [ 1833.142713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1833.363598] LustreError: 97920:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1833.445984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1835.383282] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1838.562824] LustreError: 97921:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1839.069618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1839.438071] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1841.619112] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1845.030616] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1847.687531] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1853.708381] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1853.832485] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1853.837242] LustreError: 97909:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1853.852528] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1853.856913] LustreError: 100002:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 1853.859417] LustreError: 100002:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 1853.861661] LustreError: 100002:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 1853.901824] Lustre: server umount lustre-OST0000 complete [ 1853.903973] LustreError: 100002:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1857.923773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1858.014346] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1858.018185] LustreError: 97910:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1858.023505] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1858.028140] LustreError: 100464:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 1858.031410] LustreError: 100464:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 1858.035930] LustreError: 100464:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 1858.070513] Lustre: server umount lustre-OST0000 complete [ 1858.073395] LustreError: 100464:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1862.618169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1862.732931] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1862.734996] Lustre: Found index 0 for lustre-OST0000, updating log [ 1862.738385] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 1862.768559] Lustre: lustre-OST0000: new disk, initializing [ 1862.771450] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1862.774192] Lustre: 100994:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1862.918280] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1865.814939] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1867.659637] Lustre: Failing over lustre-OST0000 [ 1867.736261] Lustre: server umount lustre-OST0000 complete [ 1870.307729] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1870.314504] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1870.321935] Lustre: Skipped 1 previous similar message [ 1870.325908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1870.329309] Lustre: Skipped 1 previous similar message [ 1874.401637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1874.401927] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1875.810893] Lustre: server umount lustre-MDT0000 complete [ 1877.554019] LustreError: 98661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789796780 with bad export cookie 14578166137505946843 [ 1877.557714] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1877.561067] LustreError: 98661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1887.810506] Lustre: server umount lustre-MDT0001 complete [ 1892.919442] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 01:46:34 (1789796794) [ 1902.604306] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 1908.021585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1908.210686] LustreError: 103338:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1908.272090] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1910.317499] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1913.315912] LustreError: 103339:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1914.731384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1917.174217] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1918.741709] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 1921.937092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1922.250735] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 1924.929322] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1925.347763] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1925.363783] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1925.363954] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1929.041982] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1929.094617] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 1929.149508] Lustre: lustre-OST0001: new disk, initializing [ 1929.153698] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1929.159211] Lustre: 105654:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1932.065941] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1935.895255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1935.900533] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1935.945655] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1941.362726] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1952.019580] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1954.274486] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 1954.278116] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1954.284782] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1954.288547] Lustre: Skipped 1 previous similar message [ 1958.937504] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.20@tcp (stopping) [ 1958.943909] Lustre: Skipped 3 previous similar messages [ 1960.228400] Lustre: server umount lustre-MDT0001 complete [ 1961.444543] LustreError: 103335:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1961.451680] LustreError: 103335:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1964.465927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1964.532964] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1964.535318] Lustre: Found index 1 for lustre-MDT0001, updating log [ 1964.557627] Lustre: 103326:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1964.938267] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1964.943476] Lustre: Skipped 3 previous similar messages [ 1966.568131] LustreError: 107360:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 1966.572141] LustreError: 107360:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 1966.576855] LustreError: 103330:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 1966.621504] LustreError: 107367:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.120@tcp: cfg command failed: rc = -17 [ 1966.626899] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 1966.626899] [ 1966.954574] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1970.156478] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1971.625070] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 01:47:52 (1789796872) [ 1980.206284] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 1990.217151] Lustre: 109266:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2001.926316] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2003.790526] Lustre: 110399:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2062.305558] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2062.305624] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2062.305887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2062.315174] Lustre: Skipped 4 previous similar messages [ 2067.683969] Lustre: server umount lustre-MDT0000 complete [ 2069.113624] LustreError: 104477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789796971 with bad export cookie 14578166137505947690 [ 2069.114894] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2069.118100] LustreError: 104477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2069.247473] Lustre: server umount lustre-MDT0001 complete [ 2080.784953] Lustre: server umount lustre-OST0000 complete [ 2091.623382] Lustre: server umount lustre-OST0001 complete [ 2094.177422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2098.778360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2102.773728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2106.794710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2114.954636] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 2119.827406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2119.837074] Lustre: lustre-MDT0000: reset Object Index mappings [ 2120.007661] LustreError: 114339:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2120.015979] LustreError: 114339:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2120.057700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2121.660917] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2125.301937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2125.310127] Lustre: lustre-MDT0001: reset Object Index mappings [ 2127.652117] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2128.832068] Lustre: 115485:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2131.816070] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2134.584020] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2136.297739] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:226 to 0x280000400:257) [ 2138.495915] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2141.382406] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2144.235624] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:195 to 0x2c0000400:225) [ 2145.265145] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 2145.266102] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:354 to 0x280000401:386) [ 2150.693776] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2152.277956] Lustre: 117361:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2172.026403] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 01:51:13 (1789797073) [ 2201.573070] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2201.574817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2201.576936] Lustre: Skipped 3 previous similar messages [ 2201.578776] Lustre: Skipped 9 previous similar messages [ 2215.392203] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2215.478716] Lustre: server umount lustre-MDT0000 complete [ 2216.931181] LustreError: 114336:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2216.937437] LustreError: 114336:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2216.974652] LustreError: 114320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789797119 with bad export cookie 14578166137505994506 [ 2216.978425] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2216.982488] LustreError: 114320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2222.052937] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2222.059708] Lustre: Skipped 1 previous similar message [ 2223.310407] Lustre: server umount lustre-MDT0001 complete [ 2230.996137] Lustre: server umount lustre-OST0000 complete [ 2238.574426] Lustre: server umount lustre-OST0001 complete [ 2240.961502] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 2245.024333] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 2260.103472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2260.202396] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2260.213679] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2260.251523] Lustre: lustre-MDT0000: new disk, initializing [ 2260.279542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2260.283275] Lustre: Skipped 3 previous similar messages [ 2260.290593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2261.844326] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2265.139631] Lustre: server umount lustre-MDT0000 complete [ 2268.944670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2269.089189] Lustre: Found index 0 for lustre-MDT0000, updating log [ 2269.098270] Lustre: 121619:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2269.107132] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 2270.837275] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2272.764797] Lustre: server umount lustre-MDT0000 complete [ 2275.945837] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 01:52:57 (1789797177) [ 2276.511943] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 2277.131088] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 01:52:58 (1789797178) [ 2283.155319] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 2286.779495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2288.408921] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2291.411170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2291.451232] Lustre: 123669:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2291.474671] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2291.476464] Lustre: Skipped 1 previous similar message [ 2291.512984] Lustre: lustre-MDT0001: new disk, initializing [ 2291.559089] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2291.563051] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2293.103659] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2295.578324] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2297.864725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2297.990856] Lustre: lustre-OST0000: new disk, initializing [ 2297.994630] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2298.001545] Lustre: 125244:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2299.839505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2299.843281] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2299.885803] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2300.226752] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2304.737126] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2304.809828] Lustre: 126115:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2306.374079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2306.378274] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2306.420300] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2307.029689] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2312.101666] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2313.672756] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2454.198061] Lustre: 132492:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 2454.200756] Lustre: 132492:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 2454.202865] Lustre: 132492:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 2454.205072] Lustre: 132492:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 2454.207228] Lustre: 132492:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 2454.208570] Lustre: 132492:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 2552.568258] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 01:57:33 (1789797453) [ 2567.650499] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2567.653254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2567.654195] Lustre: Skipped 3 previous similar messages [ 2567.656146] Lustre: Skipped 13 previous similar messages [ 2570.368532] Lustre: server umount lustre-MDT0000 complete [ 2571.610499] LustreError: 126114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789797474 with bad export cookie 14578166137506051479 [ 2571.614276] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2571.614879] LustreError: 126114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2572.772665] LustreError: 123675:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2572.781859] LustreError: 123675:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2573.815358] Lustre: server umount lustre-MDT0001 complete [ 2577.545978] Lustre: server umount lustre-OST0000 complete [ 2593.248137] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2593.317393] Lustre: server umount lustre-OST0001 complete [ 2595.573662] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 2598.945689] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 2611.553474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2611.634067] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2611.644790] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2611.647442] Lustre: Skipped 1 previous similar message [ 2611.683855] Lustre: lustre-MDT0000: new disk, initializing [ 2611.685713] Lustre: Skipped 1 previous similar message [ 2611.708446] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2611.711432] Lustre: Skipped 5 previous similar messages [ 2611.717447] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2613.122121] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2617.251299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2617.288466] Lustre: 135504:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2617.397502] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2618.789143] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2622.504636] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2624.672780] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2627.151145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2627.257370] Lustre: 137474:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2629.178591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2629.181322] Lustre: Skipped 1 previous similar message [ 2629.184202] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2629.221806] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2629.458491] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2633.370122] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2636.793425] Lustre: lustre-MDD0000: changelog on [ 2637.713654] Lustre: lustre-MDD0001: changelog on [ 3750.194433] Lustre: lustre-MDD0001: changelog off [ 3751.113401] Lustre: 135511:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 3751.601536] Lustre: lustre-MDD0000: changelog off [ 3755.489816] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3755.490394] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3755.492497] LustreError: Skipped 1 previous similar message [ 3755.497356] Lustre: Skipped 2 previous similar messages [ 3755.499487] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3755.502568] Lustre: Skipped 3 previous similar messages [ 3765.731947] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3765.734326] Lustre: Skipped 3 previous similar messages [ 3768.288131] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3768.373888] Lustre: server umount lustre-OST0000 complete [ 3770.850578] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3770.858413] Lustre: Skipped 1 previous similar message [ 3776.448789] Lustre: server umount lustre-MDT0000 complete [ 3778.285537] LustreError: 135497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789798680 with bad export cookie 14578166137506855709 [ 3778.288968] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3778.291058] LustreError: 135497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3778.477039] Lustre: server umount lustre-MDT0001 complete [ 3784.063363] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 3785.506328] Key type lgssc unregistered [ 3785.725350] LNet: 146430:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3785.731559] LNetError: 146430:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3785.740933] LNet: Removed LNI 192.168.204.120@tcp [ 3786.358148] Key type .llcrypt unregistered [ 3786.360094] Key type ._llcrypt unregistered [ 3805.423328] Key type ._llcrypt registered [ 3805.425763] Key type .llcrypt registered [ 3805.596104] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 3827.187476] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 3829.269426] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3829.403603] alg: No test for adler32 (adler32-zlib) [ 3830.556682] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [ 3830.819937] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 3832.488280] Key type lgssc registered [ 3834.059504] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3863.176606] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3863.208918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3864.354295] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3864.376361] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3864.432992] Lustre: lustre-MDT0000: new disk, initializing [ 3864.487112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3864.498978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3867.852219] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3875.308518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3875.395702] Lustre: 149628:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3875.438717] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3875.442633] Lustre: Skipped 1 previous similar message [ 3875.504994] Lustre: lustre-MDT0001: new disk, initializing [ 3875.582417] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3875.609091] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3875.619516] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3878.632945] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3885.198202] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3889.219298] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3893.903589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3894.121334] Lustre: lustre-OST0000: new disk, initializing [ 3894.125618] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3894.133043] Lustre: 151595:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3894.207915] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3898.727917] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3902.544047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3902.551959] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3902.661632] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3906.229071] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3911.674162] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_module obdecho/obdecho [ 3913.187269] LustreError: 152598:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 3913.194634] LustreError: 152598:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 3914.494865] LustreError: 152694:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 3923.428463] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3923.429401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3923.437264] Lustre: Skipped 2 previous similar messages [ 3923.442500] Lustre: Skipped 1 previous similar message [ 3928.353330] Lustre: server umount lustre-MDT0000 complete [ 3930.711775] LustreError: 149621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789798833 with bad export cookie 5124204565513201048 [ 3930.713148] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3930.718502] LustreError: 149621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3930.940923] Lustre: server umount lustre-MDT0001 complete [ 3943.801237] Lustre: server umount lustre-OST0000 complete [ 3956.878548] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 3963.667168] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 3986.090417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3986.239219] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3986.260975] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3986.310733] Lustre: lustre-MDT0000: new disk, initializing [ 3986.349169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3986.363505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3988.794455] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3996.231536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3996.330601] Lustre: 156403:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3996.387653] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3996.392428] Lustre: Skipped 1 previous similar message [ 3996.454086] Lustre: lustre-MDT0001: new disk, initializing [ 3996.540038] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3996.572352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3996.583122] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3999.607209] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4006.053268] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4009.808436] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4014.431545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4014.603756] Lustre: lustre-OST0000: new disk, initializing [ 4014.607946] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4014.612816] Lustre: 158372:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4014.679868] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4019.106810] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4020.777906] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4020.786128] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4020.852178] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4025.823886] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4028.396465] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4028.531632] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4031.143928] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4031.279879] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4036.071431] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4036.071677] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4036.077158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4036.081105] Lustre: Skipped 1 previous similar message [ 4038.678165] Lustre: server umount lustre-OST0000 complete [ 4041.186852] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4041.189094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4041.199047] Lustre: Skipped 1 previous similar message [ 4041.212908] Lustre: Skipped 2 previous similar messages [ 4046.306091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4046.310105] Lustre: Skipped 1 previous similar message [ 4047.067383] Lustre: server umount lustre-MDT0000 complete [ 4049.419616] LustreError: 156397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789798952 with bad export cookie 5124204565513202007 [ 4049.423315] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4049.425406] LustreError: 156397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4049.611197] Lustre: server umount lustre-MDT0001 complete [ 4052.034702] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 02:22:33 (1789798953) [ 4053.212688] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 4054.507503] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 02:22:35 (1789798955) [ 4059.591961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4059.851814] LustreError: 160780:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4059.866172] LustreError: 160780:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4059.903221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4062.954683] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4065.249776] LustreError: 160781:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4068.905247] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4072.534668] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4077.759177] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4081.151782] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4085.460484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4089.626477] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4094.146444] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4114.982642] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.20@tcp (stopping) [ 4115.427339] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4115.438806] Lustre: Skipped 2 previous similar messages [ 4115.443503] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4116.635217] Lustre: server umount lustre-MDT0000 complete [ 4119.086936] LustreError: 161515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789799021 with bad export cookie 5124204565513202889 [ 4119.090344] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4119.097508] LustreError: 161515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4120.083114] LustreError: 160776:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4120.547697] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4124.577705] LustreError: 160776:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4124.578131] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4124.593238] LustreError: 160776:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4124.597260] Lustre: Skipped 5 previous similar messages [ 4125.387926] Lustre: server umount lustre-MDT0001 complete [ 4129.568610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4138.436768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4138.606175] LustreError: 164643:0:(llog.c:1655:llog_backup()) MGC192.168.204.120@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 4138.611422] Lustre: 164643:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.120@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 4149.217234] LustreError: 164648:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4149.228139] LustreError: 164648:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4159.458637] LustreError: 164648:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4159.466783] LustreError: 164648:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4177.419203] LustreError: 164650:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4177.431784] LustreError: 164650:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 4199.904526] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 4199.907389] Lustre: 164643:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.120@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 4212.194264] LustreError: 164648:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4212.203248] LustreError: 164648:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 23 previous similar messages [ 4240.427061] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4240.432571] Lustre: Skipped 2 previous similar messages [ 4243.139950] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4243.496540] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4248.113333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4263.906959] LustreError: 147849:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000008c158fb1 x1876739996374912/t0(0) o250->MGC192.168.204.120@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4263.919691] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 4266.526787] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4269.545468] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4270.249489] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4271.102470] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4273.879739] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4274.637898] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4285.361970] Lustre: server umount lustre-MDT0000 complete [ 4287.397870] LustreError: 160760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789799189 with bad export cookie 5124204565513209350 [ 4287.401684] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4287.405341] LustreError: 160760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4290.018754] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4290.020108] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4290.027587] Lustre: Skipped 1 previous similar message [ 4290.031730] Lustre: Skipped 1 previous similar message [ 4290.037694] LustreError: 165394:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4290.055680] LustreError: 165394:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 41 previous similar messages [ 4293.648631] Lustre: server umount lustre-MDT0001 complete [ 4296.828549] Lustre: server umount lustre-OST0000 complete [ 4324.330776] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 4345.595243] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 4399.417759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4399.590174] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4399.634133] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4399.770778] Lustre: lustre-MDT0000: new disk, initializing [ 4399.893312] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4399.900947] Lustre: Skipped 1 previous similar message [ 4399.917956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4405.207298] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4416.437655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4416.702821] Lustre: 169750:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4417.232324] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4422.040677] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4432.782470] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4439.652426] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4446.221690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4446.556366] Lustre: lustre-OST0000: new disk, initializing [ 4446.564199] Lustre: Skipped 1 previous similar message [ 4446.571452] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4446.581655] Lustre: Skipped 2 previous similar messages [ 4446.592624] Lustre: 171719:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4451.961615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4451.972160] Lustre: Skipped 1 previous similar message [ 4451.979332] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4452.097817] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4453.671724] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4464.307424] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4469.730766] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4470.040135] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4474.893468] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4475.174942] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4477.941612] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4477.942969] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4477.952565] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4477.964239] Lustre: Skipped 1 previous similar message [ 4477.990914] Lustre: Skipped 2 previous similar messages [ 4483.530425] Lustre: server umount lustre-OST0000 complete [ 4494.489989] Lustre: server umount lustre-MDT0000 complete [ 4498.411566] LustreError: 169754:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4498.441164] LustreError: 169754:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4498.941928] LustreError: 169742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789799401 with bad export cookie 5124204565513226178 [ 4498.967706] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4498.973736] LustreError: 169742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4499.442780] Lustre: server umount lustre-MDT0001 complete [ 4509.936342] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 02:30:09 (1789799409) [ 4518.255090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4524.875457] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4534.538360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4534.907975] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4534.915528] Lustre: Skipped 3 previous similar messages [ 4540.333040] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4547.818872] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4553.711165] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4561.705976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4569.315386] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4578.450172] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4592.509804] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 02:31:32 (1789799492) [ 4600.295674] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4600.300911] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4600.305666] LustreError: Skipped 1 previous similar message [ 4600.309835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4600.318306] Lustre: Skipped 3 previous similar messages [ 4600.361246] Lustre: Skipped 7 previous similar messages [ 4603.476857] Lustre: server umount lustre-OST0000 complete [ 4613.574518] Lustre: server umount lustre-MDT0000 complete [ 4617.713881] LustreError: 174865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789799520 with bad export cookie 5124204565513227060 [ 4617.727743] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4618.046903] Lustre: server umount lustre-MDT0001 complete [ 4629.309815] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 4632.151389] Key type lgssc unregistered [ 4632.476758] LNet: 177859:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4632.482644] LNetError: 177859:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4632.500100] LNet: Removed LNI 192.168.204.120@tcp [ 4633.431432] Key type .llcrypt unregistered [ 4633.433918] Key type ._llcrypt unregistered [ 4652.044917] Key type ._llcrypt registered [ 4652.047923] Key type .llcrypt registered [ 4652.157874] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 4653.237880] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4653.380451] alg: No test for adler32 (adler32-zlib) [ 4654.430783] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [ 4654.649139] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 4656.297711] Key type lgssc registered [ 4657.292322] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4665.140702] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4665.197241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4680.806862] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4685.921128] LustreError: 179100:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.120@tcp: failed processing log, type 4: rc = -110 [ 4691.041312] LustreError: 179100:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 4691.066070] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 4691.076915] LustreError: MGC192.168.204.120@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 4691.093625] LustreError: 179100:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 4691.107916] LustreError: 179100:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -22 [ 4691.125674] LustreError: 179100:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 4691.223567] Lustre: server umount lustre-OST0000 complete [ 4691.226432] LustreError: 179100:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 4701.239046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4702.693254] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4707.709278] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4712.941316] LustreError: 179599:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4712.955355] LustreError: 179599:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4716.890881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4717.315476] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4722.123971] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4730.405305] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4736.809128] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4744.088301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4744.293152] LustreError: 181416:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 4744.308511] Lustre: 181416:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.204.120@tcp: invalid local config log lustre-OST0000: rc = -22 [ 4744.516654] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4750.762268] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4759.292884] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4762.534505] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4762.538711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4762.557100] Lustre: Skipped 1 previous similar message [ 4762.579138] Lustre: Skipped 2 previous similar messages [ 4767.736540] LustreError: 180328:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4767.832884] Lustre: server umount lustre-MDT0000 complete [ 4771.509469] LustreError: 179581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789799674 with bad export cookie 10188809110760514366 [ 4771.520204] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4771.523207] LustreError: 179581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4772.834393] LustreError: 180330:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000630d458a x1876740860310272/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4772.836305] LustreError: 179596:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4772.844135] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4772.844145] Lustre: Skipped 1 previous similar message [ 4772.849217] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4772.903423] LustreError: 179596:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4773.002643] Lustre: server umount lustre-MDT0001 complete [ 4779.042682] Lustre: server umount lustre-OST0000 complete [ 4788.725687] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 4798.292378] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 4834.013083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4834.220591] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4834.243264] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4834.298350] Lustre: lustre-MDT0000: new disk, initializing [ 4834.358025] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4834.371154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4838.746689] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4849.876750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4850.006333] Lustre: 184936:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4850.119797] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4850.123992] Lustre: Skipped 1 previous similar message [ 4850.217366] Lustre: lustre-MDT0001: new disk, initializing [ 4850.310980] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4850.355026] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4850.364357] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4855.612925] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4865.886924] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4872.269645] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4879.265910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4879.589836] Lustre: lustre-OST0000: new disk, initializing [ 4879.601187] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4879.625981] Lustre: 186907:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4879.742514] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4886.003163] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4889.132917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4889.150505] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4889.298767] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4894.554309] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4899.506771] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4899.682869] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4903.807187] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4904.003461] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4910.051603] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4910.053668] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4910.055510] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4910.064284] Lustre: Skipped 1 previous similar message [ 4915.171622] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4915.191399] Lustre: Skipped 2 previous similar messages [ 4920.295372] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4920.299768] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4920.324119] Lustre: Skipped 1 previous similar message [ 4920.489358] Lustre: server umount lustre-OST0000 complete [ 4925.417534] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4930.278206] Lustre: server umount lustre-MDT0000 complete [ 4930.530544] LustreError: 187397:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4930.546199] LustreError: 187397:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4933.725182] LustreError: 184929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789799836 with bad export cookie 10188809110760515087 [ 4933.730271] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4933.739208] LustreError: 184929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4934.149477] Lustre: server umount lustre-MDT0001 complete [ 4941.927712] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 02:37:22 (1789799842) [ 4958.928163] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 4961.404357] Key type lgssc unregistered [ 4961.793111] LNet: 189853:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4961.804200] LNetError: 189853:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4961.817776] LNet: Removed LNI 192.168.204.120@tcp [ 4962.633139] Key type .llcrypt unregistered [ 4962.635102] Key type ._llcrypt unregistered [ 4979.438366] Key type ._llcrypt registered [ 4979.445803] Key type .llcrypt registered [ 4979.539695] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 4980.421545] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4980.439598] alg: No test for adler32 (adler32-zlib) [ 4981.495877] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [ 4981.710878] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 4983.352299] Key type lgssc registered [ 4984.472337] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4991.361084] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4991.372570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4992.700281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4996.496967] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5003.234939] LustreError: 191068:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5003.245456] LustreError: 191068:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5003.655734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5003.937048] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5007.249892] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5012.661681] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5017.259866] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5023.867479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5024.051350] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5024.053188] LustreError: 192931:0:(llog.c:1696:llog_backup()) MGC192.168.204.120@tcp: failed to backup log lustre-OST0000: rc = -28 [ 5024.062118] Lustre: 192931:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.204.120@tcp: can't backup local config lustre-OST0000: rc = -28 [ 5024.157056] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5029.218630] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5035.439350] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5040.099742] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5040.108711] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5040.120487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5042.659172] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5042.662048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5042.677516] Lustre: Skipped 1 previous similar message [ 5042.697343] Lustre: Skipped 2 previous similar messages [ 5043.810037] Lustre: server umount lustre-MDT0000 complete [ 5047.062537] LustreError: 192958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789799949 with bad export cookie 17054387433567768043 [ 5047.074446] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5047.081218] LustreError: 192958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5047.364184] Lustre: server umount lustre-MDT0001 complete [ 5061.379509] Lustre: server umount lustre-OST0000 complete [ 5069.022435] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 5075.796653] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 5101.786325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5101.946610] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5101.968389] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5102.035434] Lustre: lustre-MDT0000: new disk, initializing [ 5102.106607] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5102.120661] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5105.204953] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5113.181292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5113.289344] Lustre: 196455:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5113.337391] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5113.339985] Lustre: Skipped 1 previous similar message [ 5113.346001] LustreError: 197236:0:(llog.c:1696:llog_backup()) MGC192.168.204.120@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 5113.358107] LustreError: 197236:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 5113.365496] Lustre: 197236:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.120@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 5113.381449] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5113.385172] Lustre: Skipped 1 previous similar message [ 5113.457815] Lustre: lustre-MDT0001: new disk, initializing [ 5113.535794] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5113.576460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5113.585076] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5117.217740] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5124.496260] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5128.501323] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5133.730326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5133.977396] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5133.988044] Lustre: Skipped 1 previous similar message [ 5133.992197] LustreError: 198397:0:(llog.c:1696:llog_backup()) MGC192.168.204.120@tcp: failed to backup log lustre-OST0000: rc = -28 [ 5134.001186] LustreError: 198397:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 5134.007862] Lustre: 198397:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.120@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 5134.016517] Lustre: 198397:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 5134.045686] Lustre: lustre-OST0000: new disk, initializing [ 5134.050935] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5134.058482] Lustre: 198420:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5134.112579] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5139.517712] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5142.066587] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5142.074431] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5142.209642] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5148.654978] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5153.127451] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5153.347777] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5157.525815] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5157.700857] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5162.466610] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5162.481563] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5162.495829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5165.444556] Lustre: server umount lustre-OST0000 complete [ 5168.098509] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5168.105766] Lustre: Skipped 1 previous similar message [ 5168.110229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5168.112437] Lustre: Skipped 1 previous similar message [ 5173.220246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5173.228199] Lustre: Skipped 1 previous similar message [ 5174.301088] Lustre: server umount lustre-MDT0000 complete [ 5177.336787] LustreError: 196445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789800079 with bad export cookie 17054387433567768764 [ 5177.342874] LustreError: 196445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5177.344253] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5177.595825] Lustre: server umount lustre-MDT0001 complete [ 5185.412649] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 02:41:26 (1789800086) [ 5200.765232] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 5209.577979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5209.935855] LustreError: 201543:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5209.980482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5213.755062] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5215.206751] LustreError: 201544:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5220.323409] LustreError: 201543:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5220.874168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5224.765891] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5227.368615] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5233.032598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5237.988188] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5244.363787] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5244.491187] Lustre: lustre-OST0001: new disk, initializing [ 5244.494132] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5244.502561] Lustre: 203848:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5244.580143] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5244.589436] Lustre: Skipped 2 previous similar messages [ 5250.019414] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5252.150304] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5252.157508] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5252.243205] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5260.142945] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5265.417636] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5266.982147] Lustre: DEBUG MARKER: ADD OST3 [ 5271.057268] loop: module loaded [ 5271.092154] loop0: detected capacity change from 0 to 204800000 [ 5271.264342] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5271.272577] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5271.281977] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5276.344763] loop0: detected capacity change from 0 to 204800000 [ 5277.775720] Lustre: DEBUG MARKER: STOP OST3 [ 5281.123375] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5281.285990] Lustre: lustre-OST0003: new disk, initializing [ 5281.289751] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 5281.295341] Lustre: 205815:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5282.829139] Lustre: *** cfs_fail_loc=2109, val=2*** [ 5282.950447] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 5282.959487] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 5282.997809] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 5284.896363] LustreError: 206017:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 5285.030762] Lustre: Failing over lustre-OST0003 [ 5285.166924] Lustre: server umount lustre-OST0003 complete [ 5286.637071] LustreError: 203031:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.204.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5286.829612] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5287.921857] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5287.945590] Lustre: Skipped 1 previous similar message [ 5295.457652] Lustre: DEBUG MARKER: START OST3 again [ 5297.170508] LustreError: 204315:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.204.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5297.191789] LustreError: 204315:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 5300.958033] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5301.216213] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 5302.286296] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 5303.036594] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5303.037057] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 5306.478789] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5325.014964] Lustre: Failing over lustre-OST0003 [ 5325.102842] Lustre: server umount lustre-OST0003 complete [ 5326.829638] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5326.831624] LustreError: 203031:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5326.836878] Lustre: Skipped 2 previous similar messages [ 5326.852196] LustreError: 203031:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5342.187265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5342.196114] Lustre: Skipped 3 previous similar messages [ 5346.229841] Lustre: server umount lustre-MDT0000 complete [ 5348.883027] LustreError: 201524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789800251 with bad export cookie 17054387433567769639 [ 5348.888354] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5348.906550] LustreError: 201524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5352.417627] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5352.424211] Lustre: Skipped 5 previous similar messages [ 5355.308151] Lustre: server umount lustre-MDT0001 complete [ 5358.687552] Lustre: server umount lustre-OST0000 complete [ 5362.132304] Lustre: server umount lustre-OST0001 complete [ 5368.006091] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 5374.592163] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 5401.581097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5401.803142] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5401.823355] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5401.876357] Lustre: lustre-MDT0000: new disk, initializing [ 5401.926716] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5401.932918] Lustre: Skipped 2 previous similar messages [ 5401.943977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5405.815411] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5415.195301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5415.313806] Lustre: 211303:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5415.641064] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5419.134493] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5426.726132] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5431.069221] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5435.940120] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5436.188718] Lustre: lustre-OST0000: new disk, initializing [ 5436.195103] Lustre: Skipped 1 previous similar message [ 5436.200311] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5436.206707] Lustre: Skipped 2 previous similar messages [ 5436.211828] Lustre: 213270:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5437.565225] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5437.571727] Lustre: Skipped 1 previous similar message [ 5437.578510] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5437.675866] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5441.218805] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5448.366689] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5452.140455] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5452.293369] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5455.687743] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5455.834744] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5458.410376] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5458.412288] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5458.432933] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5458.433015] LustreError: Skipped 2 previous similar messages [ 5458.448501] Lustre: Skipped 5 previous similar messages [ 5463.454148] Lustre: server umount lustre-OST0000 complete [ 5467.106086] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5472.246850] Lustre: server umount lustre-MDT0000 complete [ 5473.762782] LustreError: 211314:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5473.777770] LustreError: 211314:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 5474.852694] LustreError: 211297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789800377 with bad export cookie 17054387433567771942 [ 5474.856419] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5474.861685] LustreError: 211297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5475.123905] Lustre: server umount lustre-MDT0001 complete [ 5477.956926] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 02:46:18 (1789800378) [ 5491.138973] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 5500.728302] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 5530.746337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5530.935350] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5530.956153] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5531.017473] Lustre: lustre-MDT0000: new disk, initializing [ 5531.105847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5531.114544] Lustre: Skipped 2 previous similar messages [ 5531.128526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5534.836097] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5543.176084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5543.275961] Lustre: 217885:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5543.515514] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5547.166816] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5554.542709] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5558.970752] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5563.977370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5564.193116] Lustre: 219851:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5566.124177] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5569.000512] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5575.995900] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5578.881121] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5579.044714] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5582.234586] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5582.373661] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5586.401499] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5586.405883] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5586.414536] Lustre: Skipped 3 previous similar messages [ 5586.420323] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5586.424968] Lustre: Skipped 3 previous similar messages [ 5598.180483] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5598.295833] Lustre: server umount lustre-OST0000 complete [ 5607.248443] Lustre: server umount lustre-MDT0000 complete [ 5607.399923] LustreError: 217897:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5607.416249] LustreError: 217897:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5610.042241] LustreError: 217878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789800512 with bad export cookie 17054387433567772824 [ 5610.052232] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5610.056698] LustreError: 217878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5610.320726] Lustre: server umount lustre-MDT0001 complete [ 5616.161695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5619.870837] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5625.419272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5628.544062] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5633.280853] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5636.847313] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5641.545991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5645.469124] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5650.306309] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5658.595504] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5658.603202] LustreError: Skipped 1 previous similar message [ 5658.611128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5658.614929] Lustre: Skipped 7 previous similar messages [ 5664.666957] Lustre: server umount lustre-OST0000 complete [ 5673.678206] Lustre: server umount lustre-MDT0000 complete [ 5675.934972] LustreError: 222050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789800578 with bad export cookie 17054387433567773706 [ 5675.937169] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5675.941931] LustreError: 222050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5676.134103] Lustre: server umount lustre-MDT0001 complete [ 5787.975953] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 5789.770664] Key type lgssc unregistered [ 5789.948068] LNet: 225732:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5789.954160] LNetError: 225732:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5789.963131] LNet: Removed LNI 192.168.204.120@tcp [ 5790.502264] Key type .llcrypt unregistered [ 5790.504289] Key type ._llcrypt unregistered [ 5806.373970] Key type ._llcrypt registered [ 5806.376348] Key type .llcrypt registered [ 5806.454815] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 5816.629939] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 5817.204950] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5817.238082] alg: No test for adler32 (adler32-zlib) [ 5818.230655] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [ 5818.401565] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 5820.048197] Key type lgssc registered [ 5820.756432] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5842.797053] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5842.811421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5843.973411] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5843.994717] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5844.052393] Lustre: lustre-MDT0000: new disk, initializing [ 5844.097640] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5844.116484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5847.210451] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5854.826920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5854.898116] Lustre: 229056:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5854.956924] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5854.961750] Lustre: Skipped 1 previous similar message [ 5855.034370] Lustre: lustre-MDT0001: new disk, initializing [ 5855.115288] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5855.144762] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5855.153569] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5858.125320] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5864.832351] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5869.046400] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5873.697526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5873.914479] Lustre: lustre-OST0000: new disk, initializing [ 5873.917738] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5873.924536] Lustre: 231023:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5873.978537] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5878.126339] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5880.883433] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5880.888472] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5880.982757] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5884.943963] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5888.147897] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5888.285298] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5891.188185] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5891.323921] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5896.163782] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5896.169548] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5896.180889] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5896.676592] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5896.676925] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5896.691387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5898.764547] Lustre: server umount lustre-OST0000 complete [ 5901.284140] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5901.288215] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5901.297724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5906.915250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5906.919277] Lustre: Skipped 2 previous similar messages [ 5907.479676] Lustre: server umount lustre-MDT0000 complete [ 5910.191319] LustreError: 229046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789800812 with bad export cookie 8658619705186127741 [ 5910.197361] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5910.201317] LustreError: 229046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5910.460833] Lustre: server umount lustre-MDT0001 complete [ 5924.152989] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 5930.469493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5930.753035] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5933.514123] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5936.103382] LustreError: 233957:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5936.114659] LustreError: 233957:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5939.423535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5939.682042] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5943.126734] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5945.262416] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5950.074798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5950.288866] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5953.948831] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5959.851190] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5959.980549] Lustre: lustre-OST0001: new disk, initializing [ 5959.985240] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5959.991646] Lustre: 236262:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5964.377159] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5964.874808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5964.881575] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5964.945980] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5974.390845] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5986.074755] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5996.290842] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 02:54:56 (1789800896) [ 6001.122517] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6001.131848] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6001.146969] Lustre: Skipped 1 previous similar message [ 6001.154511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6012.389217] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6012.391497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6012.402283] Lustre: Skipped 11 previous similar messages [ 6012.524855] Lustre: server umount lustre-MDT0000 complete [ 6014.817546] LustreError: 233937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789800917 with bad export cookie 8658619705186128623 [ 6014.820720] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6014.831767] LustreError: 233937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6015.033282] Lustre: server umount lustre-MDT0001 complete [ 6028.003835] Lustre: server umount lustre-OST0000 complete [ 6039.782687] Lustre: server umount lustre-OST0001 complete [ 6044.646708] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 6051.977471] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 6077.033575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6077.190818] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6077.221717] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6077.282642] Lustre: lustre-MDT0000: new disk, initializing [ 6077.327776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6077.333261] Lustre: Skipped 1 previous similar message [ 6077.343911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6080.251438] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6087.717688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6087.795287] Lustre: 240209:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6087.856156] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6087.863112] Lustre: Skipped 1 previous similar message [ 6087.936915] Lustre: lustre-MDT0001: new disk, initializing [ 6088.048670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6088.064740] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6090.756964] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6096.623920] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6100.421550] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6104.805219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6104.999859] Lustre: lustre-OST0000: new disk, initializing [ 6105.003279] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6105.008405] Lustre: 242176:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6106.717034] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6106.723996] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6106.803340] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6108.972107] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6115.315276] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6118.237475] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6118.357618] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6121.385604] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6121.609373] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6127.587079] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6127.592100] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6127.597316] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6127.609175] Lustre: Skipped 4 previous similar messages [ 6137.312376] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6137.422441] Lustre: server umount lustre-OST0000 complete [ 6142.946785] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6145.940925] Lustre: server umount lustre-MDT0000 complete [ 6148.065933] LustreError: 241009:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6148.086331] LustreError: 241009:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6148.410549] LustreError: 240201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789801050 with bad export cookie 8658619705186130758 [ 6148.414315] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6148.422610] LustreError: 240201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6148.663183] Lustre: server umount lustre-MDT0001 complete [ 6154.013346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6154.304424] LustreError: 244392:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6154.351057] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6154.356931] Lustre: Skipped 2 previous similar messages [ 6157.073295] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6159.846503] LustreError: 244392:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6162.565460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6165.695254] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6170.418527] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6173.771522] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6178.484583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6182.703791] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6187.643426] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6333.409245] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6333.418472] LustreError: Skipped 1 previous similar message [ 6333.421894] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6333.429399] Lustre: Skipped 1 previous similar message [ 6333.434711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6333.440688] Lustre: Skipped 5 previous similar messages [ 6339.314221] Lustre: server umount lustre-OST0000 complete [ 6347.927736] Lustre: server umount lustre-MDT0000 complete [ 6350.528702] LustreError: 244373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789801253 with bad export cookie 8658619705186131640 [ 6350.534410] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6350.537571] LustreError: 244373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6350.734206] Lustre: server umount lustre-MDT0001 complete [ 6455.004389] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 6456.608504] Key type lgssc unregistered [ 6456.769400] LNet: 248007:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6456.774330] LNetError: 248007:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6456.785629] LNet: Removed LNI 192.168.204.120@tcp [ 6457.289163] Key type .llcrypt unregistered [ 6457.291160] Key type ._llcrypt unregistered [ 6470.875272] Key type ._llcrypt registered [ 6470.876920] Key type .llcrypt registered [ 6470.929201] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 6479.420207] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 6480.020566] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6480.035120] alg: No test for adler32 (adler32-zlib) [ 6480.955402] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [ 6481.090498] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 6482.704129] Key type lgssc registered [ 6483.257509] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6507.526737] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 6512.730951] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6512.747381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6513.874381] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6513.894535] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6513.946096] Lustre: lustre-MDT0000: new disk, initializing [ 6513.990188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6514.005594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6516.688258] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6523.524861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6523.594941] Lustre: 252048:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6523.634990] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6523.638099] Lustre: Skipped 1 previous similar message [ 6523.694285] Lustre: lustre-MDT0001: new disk, initializing [ 6523.752028] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6523.776148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6523.786657] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6526.160811] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6529.389686] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6533.123246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6533.296657] Lustre: lustre-OST0000: new disk, initializing [ 6533.299560] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6533.304984] Lustre: 253677:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6533.361032] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6536.527061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6536.536735] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6536.618626] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6536.857141] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6543.070750] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6543.162220] Lustre: lustre-OST0001: new disk, initializing [ 6543.164197] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6543.167697] Lustre: 254548:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6543.223255] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6544.495693] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6544.504443] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6544.561435] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6546.283789] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6554.332485] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6558.129432] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6564.833643] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6564.837626] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6564.856447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6565.349743] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6565.355671] Lustre: Skipped 1 previous similar message [ 6565.358667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6565.362043] Lustre: Skipped 3 previous similar messages [ 6570.406205] Lustre: server umount lustre-MDT0000 complete [ 6572.629848] LustreError: 252040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789801475 with bad export cookie 758584548960405923 [ 6572.637519] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6572.638946] LustreError: 252040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6572.816230] Lustre: server umount lustre-MDT0001 complete [ 6584.866934] Lustre: server umount lustre-OST0000 complete [ 6597.253443] Lustre: server umount lustre-OST0001 complete [ 6605.356454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6611.345933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6616.134724] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6621.614958] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6632.120238] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 6637.359498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6637.375985] Lustre: lustre-MDT0000: reset Object Index mappings [ 6637.632241] LustreError: 259555:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6637.645368] LustreError: 259555:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6637.684626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6640.068615] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6643.170371] LustreError: 259556:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6644.704864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6644.713589] Lustre: lustre-MDT0001: reset Object Index mappings [ 6647.001241] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6648.334415] Lustre: 260697:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6651.610613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6651.803283] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6651.806553] Lustre: Skipped 1 previous similar message [ 6654.887527] LustreError: 261050:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6654.973244] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6659.526050] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6662.769503] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6662.770596] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 6662.864702] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6667.226140] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6668.891062] Lustre: 262566:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6673.385576] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6673.387498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6673.399932] Lustre: Skipped 3 previous similar messages [ 6673.407484] Lustre: Skipped 3 previous similar messages [ 6677.017325] Lustre: server umount lustre-MDT0000 complete [ 6678.498337] LustreError: 259555:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6678.512878] LustreError: 259555:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6678.925466] LustreError: 259536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789801581 with bad export cookie 758584548960412713 [ 6678.929266] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6678.930577] LustreError: 259536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6679.083711] Lustre: server umount lustre-MDT0001 complete [ 6690.833307] Lustre: server umount lustre-OST0000 complete [ 6702.779915] Lustre: server umount lustre-OST0001 complete [ 6712.559992] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 03:06:53 (1789801613) [ 6720.305360] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 6725.428824] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 6742.632288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6742.757283] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6742.776625] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6742.819116] Lustre: lustre-MDT0000: new disk, initializing [ 6742.857963] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6742.861662] Lustre: Skipped 1 previous similar message [ 6742.871438] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6744.863172] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6750.561188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6750.620950] Lustre: 266813:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6750.779828] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6752.736523] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6758.519109] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6761.145258] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6764.303142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6764.442468] Lustre: lustre-OST0000: new disk, initializing [ 6764.444270] Lustre: Skipped 1 previous similar message [ 6764.446447] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6764.449028] Lustre: Skipped 2 previous similar messages [ 6764.452080] Lustre: 268784:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6765.638575] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6765.642720] Lustre: Skipped 1 previous similar message [ 6765.645429] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6765.702713] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6767.149292] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6772.148456] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6774.279593] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6774.380631] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6776.257113] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6776.359141] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6781.410235] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6781.410843] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6781.414048] LustreError: Skipped 1 previous similar message [ 6781.419788] Lustre: Skipped 1 previous similar message [ 6781.423098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6781.427296] Lustre: Skipped 1 previous similar message [ 6783.328396] Lustre: server umount lustre-OST0000 complete [ 6786.530547] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6786.534911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6786.538460] Lustre: Skipped 1 previous similar message [ 6786.544829] Lustre: Skipped 1 previous similar message [ 6796.769148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6796.771890] Lustre: Skipped 3 previous similar messages [ 6799.328118] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6799.432337] Lustre: server umount lustre-MDT0000 complete [ 6801.144791] LustreError: 267609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789801703 with bad export cookie 758584548960414799 [ 6801.147879] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6801.150730] LustreError: 267609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6801.330828] Lustre: server umount lustre-MDT0001 complete [ 6809.687950] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 6813.829206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6813.996286] LustreError: 271711:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6814.015997] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6814.019559] Lustre: Skipped 2 previous similar messages [ 6815.714039] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6819.244039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6821.160398] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6822.607320] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6825.744538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6828.568714] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6832.588670] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6832.673525] Lustre: lustre-OST0001: new disk, initializing [ 6832.676501] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6832.681539] Lustre: 274017:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6833.813977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6833.818145] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6833.853184] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6835.242852] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6840.571077] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6842.196385] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6844.764802] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 6844.795189] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 6849.510574] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6849.511795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6849.515707] Lustre: Skipped 2 previous similar messages [ 6849.519453] Lustre: Skipped 3 previous similar messages [ 6853.482421] Lustre: server umount lustre-MDT0000 complete [ 6854.626297] LustreError: 271708:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6854.633146] LustreError: 271708:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6854.908234] LustreError: 271694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789801757 with bad export cookie 758584548960415681 [ 6854.912595] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6854.913210] LustreError: 271694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6855.048708] Lustre: server umount lustre-MDT0001 complete [ 6866.973152] Lustre: server umount lustre-OST0000 complete [ 6879.110429] Lustre: server umount lustre-OST0001 complete [ 6886.632166] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 6890.925960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6891.083620] LustreError: 277069:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6891.106754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6891.109610] Lustre: Skipped 3 previous similar messages [ 6892.717312] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6896.355785] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6898.458915] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6899.789442] Lustre: 278210:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6902.765210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6905.625967] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6909.508972] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6912.066155] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6916.087214] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 6916.099033] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 6921.104797] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6922.767929] Lustre: 280086:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6925.187053] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 6925.191555] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 6929.498425] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 03:10:30 (1789801830) [ 6941.666345] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6941.668347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6941.673331] Lustre: Skipped 3 previous similar messages [ 6941.676082] Lustre: Skipped 1 previous similar message [ 6943.490435] Lustre: server umount lustre-MDT0000 complete [ 6945.056318] LustreError: 280088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789801847 with bad export cookie 758584548960421995 [ 6945.059454] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6945.061185] LustreError: 280088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6946.784989] LustreError: 280382:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6946.791416] LustreError: 280382:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6947.235835] Lustre: server umount lustre-MDT0001 complete [ 6950.985043] Lustre: server umount lustre-OST0000 complete [ 6958.702489] Lustre: server umount lustre-OST0001 complete [ 6961.461407] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 6965.643786] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 6984.971759] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 6988.909935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6989.006357] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6989.019577] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6989.057943] Lustre: lustre-MDT0000: new disk, initializing [ 6989.098840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6990.782760] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6995.718646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6995.768678] Lustre: 284317:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6995.898956] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6997.643163] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7003.426179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7003.564339] Lustre: 285953:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7004.938988] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7006.290218] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7011.786256] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7011.875188] Lustre: 286822:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7013.709743] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7014.519650] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7019.843683] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7021.409539] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7021.412665] Lustre: Skipped 1 previous similar message [ 7026.885845] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 03:12:08 (1789801928) [ 7029.220477] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7029.223273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7029.227262] Lustre: Skipped 5 previous similar messages [ 7029.230089] Lustre: Skipped 5 previous similar messages [ 7033.961536] Lustre: server umount lustre-MDT0000 complete [ 7034.339197] LustreError: 284325:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7034.346591] LustreError: 284325:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7035.418990] LustreError: 284311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789801938 with bad export cookie 758584548960429877 [ 7035.422654] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7035.424779] LustreError: 284311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7035.550569] Lustre: server umount lustre-MDT0001 complete [ 7046.672246] Lustre: server umount lustre-OST0000 complete [ 7058.482479] Lustre: server umount lustre-OST0001 complete [ 7062.224064] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7062.485206] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7062.488123] Lustre: Skipped 7 previous similar messages [ 7064.292500] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7067.939569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7069.808067] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7072.829870] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7075.030284] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7077.970587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7080.571147] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7083.669561] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7096.818103] loop1: detected capacity change from 0 to 204800000 [ 7096.907590] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7096.916214] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7096.926520] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7099.359917] loop1: detected capacity change from 0 to 204800000 [ 7101.839893] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7101.885059] LustreError: 289053:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs2414: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 7101.893091] LustreError: 289053:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs2414 [ 7101.895892] LustreError: 289053:0:(mgs_handler.c:517:mgs_target_reg()) fs2414-MDT0001: Error checking target index: rc = -13 [ 7101.901529] LustreError: fs2414-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 7101.907304] LustreError: 292781:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [ 7101.911185] LustreError: 292781:0:(tgt_mount.c:2132:server_put_super()) no obd fs2414-MDT0001 [ 7101.915501] LustreError: 292781:0:(tgt_mount.c:116:server_deregister_mount()) fs2414-MDT0001 not registered [ 7101.931626] Lustre: server umount fs2414-MDT0001 complete [ 7101.934475] LustreError: 292781:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 7105.588690] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7105.635841] Lustre: Setting parameter fs2414-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs2414-MDT0001 [ 7105.662956] Lustre: srv-fs2414-MDT0001: No data found on store. Initialize space. [ 7105.667707] Lustre: Skipped 4 previous similar messages [ 7105.714314] Lustre: fs2414-MDT0001: new disk, initializing [ 7105.717509] Lustre: Skipped 3 previous similar messages [ 7105.764507] Lustre: srv-fs2414-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 7105.767705] Lustre: 293248:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs2414-MDT0001-mdtlov: can't create new llog: rc = -115 [ 7105.772053] LustreError: 293248:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs2414-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 7107.531349] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7110.883176] Lustre: server umount fs2414-MDT0001 complete [ 7116.481920] loop1: detected capacity change from 0 to 204800000 [ 7118.810704] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7118.929176] Lustre: 294813:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs2414-MDT0001-mdtlov: can't create new llog: rc = -115 [ 7118.933077] LustreError: 294813:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs2414-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 7120.371790] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7122.186656] Lustre: server umount fs2414-MDT0001 complete [ 7128.549458] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7128.554442] Lustre: Skipped 2 previous similar messages [ 7140.320198] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7140.408394] Lustre: server umount lustre-MDT0000 complete [ 7144.366707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7144.436661] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7146.433285] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7149.801402] loop1: detected capacity change from 0 to 204800000 [ 7150.060888] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7152.262169] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7152.421906] Lustre: 297256:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs2414-MDT0001-mdtlov: can't create new llog: rc = -115 [ 7152.427284] LustreError: 297256:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs2414-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 7154.279301] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7156.213804] Lustre: server umount fs2414-MDT0001 complete [ 7164.897326] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7164.901107] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7164.902954] Lustre: Skipped 13 previous similar messages [ 7165.411960] LustreError: 291565:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7165.418256] LustreError: 291565:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 51 previous similar messages [ 7170.201918] Lustre: server umount lustre-OST0000 complete [ 7175.650074] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7175.653365] LustreError: Skipped 1 previous similar message [ 7177.820280] Lustre: server umount lustre-MDT0000 complete [ 7179.227299] LustreError: 295303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789802081 with bad export cookie 758584548960432551 [ 7179.229542] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7179.232406] LustreError: 295303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7179.365866] Lustre: server umount lustre-MDT0001 complete [ 7183.519061] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 7184.744211] Key type lgssc unregistered [ 7184.891564] LNet: 299593:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7184.895044] LNetError: 299593:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7184.903385] LNet: Removed LNI 192.168.204.120@tcp [ 7185.290195] Key type .llcrypt unregistered [ 7185.291412] Key type ._llcrypt unregistered [ 7193.595193] Key type ._llcrypt registered [ 7193.596476] Key type .llcrypt registered [ 7193.643374] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 7194.004704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7194.055133] alg: No test for adler32 (adler32-zlib) [ 7194.921945] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [ 7195.022105] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 7196.608217] Key type lgssc registered [ 7197.005430] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7200.024196] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7200.034499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7201.226424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7202.854820] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7206.305926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7206.490652] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7208.085789] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7210.933886] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7213.031920] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7215.584643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7215.731137] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7216.742192] LustreError: 302666:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7216.750281] LustreError: 302666:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7218.114857] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7221.121197] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7222.241984] LustreError: 302665:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7226.341375] LustreError: 302820:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7226.349350] LustreError: 302820:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7231.463755] LustreError: 302666:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7231.470216] LustreError: 302666:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7232.943463] loop1: detected capacity change from 0 to 204800000 [ 7233.053120] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7233.057664] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7233.063480] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7234.958118] loop1: detected capacity change from 0 to 204800000 [ 7236.580995] LustreError: 302666:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7236.587550] LustreError: 302666:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7237.075768] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7237.133566] Lustre: fs2978-OST0000: new disk, initializing [ 7237.135458] Lustre: srv-fs2978-OST0000: No data found on store. Initialize space. [ 7237.138419] Lustre: 304571:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7237.170632] Lustre: fs2978-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7239.542635] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7243.628440] Lustre: server umount fs2978-OST0000 complete [ 7246.818765] LustreError: 302665:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7246.825737] LustreError: 302665:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 7248.732192] loop1: detected capacity change from 0 to 204800000 [ 7250.896732] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7250.977259] Lustre: fs2978-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7253.191287] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7256.550398] loop2: detected capacity change from 0 to 204800000 [ 7256.616733] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7256.622537] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7256.627742] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7258.779197] loop2: detected capacity change from 0 to 204800000 [ 7261.026030] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7261.062634] LustreError: fs2978-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 7261.067920] LustreError: fs2978-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 7261.072815] LustreError: 307609:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [ 7261.075675] LustreError: 307609:0:(tgt_mount.c:2132:server_put_super()) no obd fs2978-OST0001 [ 7261.078633] LustreError: 307609:0:(tgt_mount.c:116:server_deregister_mount()) fs2978-OST0001 not registered [ 7261.097125] Lustre: server umount fs2978-OST0001 complete [ 7261.099102] LustreError: 307609:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 7265.004103] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7265.054889] Lustre: fs2978-OST0001: new disk, initializing [ 7265.057512] Lustre: srv-fs2978-OST0001: No data found on store. Initialize space. [ 7265.061227] Lustre: 308160:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7265.093626] Lustre: fs2978-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7267.301164] LustreError: 302819:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7267.305903] LustreError: 302819:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 7267.390976] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7271.308203] Lustre: server umount fs2978-OST0000 complete [ 7275.188714] Lustre: server umount fs2978-OST0001 complete [ 7279.689344] loop1: detected capacity change from 0 to 204800000 [ 7281.670357] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7281.744845] Lustre: fs2978-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7283.912872] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7286.378522] loop2: detected capacity change from 0 to 204800000 [ 7288.499734] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7290.860283] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7294.278408] Lustre: server umount fs2978-OST0000 complete [ 7297.864245] Lustre: server umount fs2978-OST0001 complete [ 7303.139530] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7303.140411] LustreError: 302665:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7303.142231] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7303.145038] Lustre: Skipped 1 previous similar message [ 7303.151355] LustreError: 302665:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 20 previous similar messages [ 7303.154656] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7308.259762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7308.262496] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [ 7308.264668] Lustre: Skipped 2 previous similar messages [ 7308.521978] Lustre: server umount lustre-OST0000 complete [ 7313.377466] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7313.379572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7313.382415] Lustre: Skipped 1 previous similar message [ 7316.142457] Lustre: server umount lustre-MDT0000 complete [ 7317.497320] LustreError: 300819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789802220 with bad export cookie 4623222244273266992 [ 7317.500664] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7317.502470] LustreError: 300819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7317.638860] Lustre: server umount lustre-MDT0001 complete [ 7321.530489] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 7322.647662] Key type lgssc unregistered [ 7322.773424] LNet: 314070:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7322.777212] LNetError: 314070:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7322.786302] LNet: Removed LNI 192.168.204.120@tcp [ 7323.131673] Key type .llcrypt unregistered [ 7323.133266] Key type ._llcrypt unregistered [ 7334.334203] Key type ._llcrypt registered [ 7334.335683] Key type .llcrypt registered [ 7334.386913] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 7340.916403] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 7341.342415] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7341.353935] alg: No test for adler32 (adler32-zlib) [ 7342.211590] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [ 7342.312501] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 7343.896137] Key type lgssc registered [ 7344.268713] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7357.080160] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7357.090067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7358.192358] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7358.211370] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7358.257109] Lustre: lustre-MDT0000: new disk, initializing [ 7358.293212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7358.302449] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7359.839069] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7364.313535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7364.361119] Lustre: 317896:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7364.394524] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7364.396845] Lustre: Skipped 1 previous similar message [ 7364.431761] Lustre: lustre-MDT0001: new disk, initializing [ 7364.469888] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7364.486622] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7364.491270] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7365.997395] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7369.953454] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7371.970933] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7374.676532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7374.786248] Lustre: lustre-OST0000: new disk, initializing [ 7374.788114] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7374.792188] Lustre: 319866:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7374.827364] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7377.077364] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7380.989758] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7380.993694] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7381.029131] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7381.279818] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7382.053071] Lustre: Mounted lustre-client - version 2.17.57_110_g5f5d390 [ 7382.408985] Lustre: Unmounted lustre-client [ 7386.081230] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7386.081460] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7386.084432] LustreError: Skipped 1 previous similar message [ 7386.090452] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7386.094185] Lustre: Skipped 1 previous similar message [ 7387.109952] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7387.112412] Lustre: Skipped 1 previous similar message [ 7387.732948] Lustre: server umount lustre-OST0000 complete [ 7390.176903] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7390.179944] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7390.184594] Lustre: Skipped 1 previous similar message [ 7390.186751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7392.227873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7392.228706] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7395.235626] Lustre: server umount lustre-MDT0000 complete [ 7396.606185] LustreError: 318690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789802299 with bad export cookie 10823570301842774920 [ 7396.609222] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7396.611184] LustreError: 318690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7396.728090] Lustre: server umount lustre-MDT0001 complete [ 7400.507629] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 7401.608915] Key type lgssc unregistered [ 7434.465069] LNet: 1 local NIs in recovery (showing 1): 192.168.204.120@tcp [ 7434.528899] LNet: 322020:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7434.532954] LNetError: 322020:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7513.382376] LNet: Removed LNI 192.168.204.120@tcp [ 7513.720072] Key type .llcrypt unregistered [ 7513.721366] Key type ._llcrypt unregistered [ 7519.885584] Key type ._llcrypt registered [ 7519.886582] Key type .llcrypt registered [ 7519.927730] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 7526.584476] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 7526.926223] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7526.998271] alg: No test for adler32 (adler32-zlib) [ 7527.858456] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [ 7527.951402] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 7529.536307] Key type lgssc registered [ 7529.947057] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7540.492360] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 03:20:42 (1789802442) [ 7542.847496] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7542.853230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7543.945749] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7543.957633] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7543.995159] Lustre: lustre-MDT0000: new disk, initializing [ 7544.025386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7544.033438] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7545.477104] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7549.701177] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7549.742462] Lustre: 325215:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7549.766777] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7549.768910] Lustre: Skipped 1 previous similar message [ 7549.803876] Lustre: lustre-MDT0001: new disk, initializing [ 7549.834982] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7549.847727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7549.851357] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7551.147953] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7554.898896] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7556.787457] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7559.047454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7559.146621] Lustre: lustre-OST0000: new disk, initializing [ 7559.148897] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7559.152275] Lustre: 327183:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7559.177951] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7561.111932] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7564.871817] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7568.379467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7568.383020] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7568.413142] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7578.595167] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7578.595217] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7578.596481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7578.598200] LustreError: Skipped 1 previous similar message [ 7578.603623] Lustre: Skipped 1 previous similar message [ 7582.100858] Lustre: server umount lustre-OST0000 complete [ 7583.714141] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7583.714784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7583.718557] Lustre: Skipped 1 previous similar message [ 7583.723175] Lustre: Skipped 2 previous similar messages [ 7588.833156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7588.836560] Lustre: Skipped 1 previous similar message [ 7589.595859] Lustre: server umount lustre-MDT0000 complete [ 7590.856242] LustreError: 325207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789802493 with bad export cookie 18240840231302377254 [ 7590.859854] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7590.860786] LustreError: 325207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7590.975804] Lustre: server umount lustre-MDT0001 complete [ 7594.901816] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 7596.069975] Key type lgssc unregistered [ 7596.202548] LNet: 329089:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7596.205675] LNetError: 329089:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7596.220391] LNet: Removed LNI 192.168.204.120@tcp [ 7596.586385] Key type .llcrypt unregistered [ 7596.588539] Key type ._llcrypt unregistered [ 7605.204209] Key type ._llcrypt registered [ 7605.206448] Key type .llcrypt registered [ 7605.248584] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 7605.640846] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7605.657417] alg: No test for adler32 (adler32-zlib) [ 7606.522453] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [ 7606.614537] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 7608.192193] Key type lgssc registered [ 7608.585612] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7611.703543] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7611.710201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7612.863964] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7614.367796] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7617.512373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7617.661867] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7619.090024] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7621.631602] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7623.700172] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7626.259913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7626.389366] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7628.471071] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7631.164083] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7644.934375] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7645.428752] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7646.738938] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.20@tcp (stopping) [ 7648.225684] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7648.226601] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7648.228477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7648.233901] Lustre: Skipped 2 previous similar messages [ 7651.850256] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.20@tcp (stopping) [ 7651.853142] Lustre: Skipped 2 previous similar messages [ 7656.970339] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.20@tcp (stopping) [ 7656.972841] Lustre: Skipped 3 previous similar messages [ 7661.024116] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7661.090204] Lustre: server umount lustre-MDT0000 complete [ 7662.090373] LustreError: 330333:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7662.096330] LustreError: 330333:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7662.368105] LustreError: 330320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789802564 with bad export cookie 10816742753526762842 [ 7662.371340] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7662.372157] LustreError: 330320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7663.584958] LustreError: 332338:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7663.585118] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7663.593843] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7663.595726] Lustre: Skipped 3 previous similar messages [ 7667.210166] LustreError: 330332:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7668.522950] Lustre: server umount lustre-MDT0001 complete [ 7669.780249] Lustre: server umount lustre-OST0000 complete [ 7672.520404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7673.788901] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7674.663594] Lustre: 334533:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.120@tcp [ 7675.323874] Lustre: server umount MGS complete [ 7678.040255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7678.191524] LustreError: 335105:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7678.198234] LustreError: 335105:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7678.213375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7679.530244] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7682.426363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7682.559113] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7683.971669] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7686.861569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7688.965285] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7691.545681] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7692.128539] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [ 7693.946836] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7698.562732] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 7703.522170] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7703.522675] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7703.523275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7703.523279] Lustre: Skipped 2 previous similar messages [ 7703.524938] LustreError: Skipped 1 previous similar message [ 7703.528568] Lustre: Skipped 1 previous similar message [ 7705.735435] Lustre: server umount lustre-OST0000 complete [ 7708.128832] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7708.131801] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7713.178882] Lustre: server umount lustre-MDT0000 complete [ 7713.761544] LustreError: 335105:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7713.766561] LustreError: 335105:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7714.458275] LustreError: 335840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789802617 with bad export cookie 10816742753526763668 [ 7714.461572] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7714.462382] LustreError: 335840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7714.581626] Lustre: server umount lustre-MDT0001 complete [ 7718.460791] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 7719.549360] Key type lgssc unregistered [ 7719.685449] LNet: 338538:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7719.688289] LNetError: 338538:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7719.695646] LNet: Removed LNI 192.168.204.120@tcp [ 7720.030944] Key type .llcrypt unregistered [ 7720.032071] Key type ._llcrypt unregistered [ 7726.284691] Key type ._llcrypt registered [ 7726.285543] Key type .llcrypt registered [ 7726.321350] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 7732.911525] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 7733.366967] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7733.399896] alg: No test for adler32 (adler32-zlib) [ 7734.265359] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [ 7734.373142] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 7735.960203] Key type lgssc registered [ 7736.369649] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7748.616660] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7748.623529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7749.729148] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7749.741410] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7749.783409] Lustre: lustre-MDT0000: new disk, initializing [ 7749.811810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7749.818450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7751.287893] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7755.546062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7755.588796] Lustre: 341666:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7755.618389] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7755.620881] Lustre: Skipped 1 previous similar message [ 7755.655214] Lustre: lustre-MDT0001: new disk, initializing [ 7755.693993] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7755.710290] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7755.714374] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7757.212544] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7761.113906] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7762.997839] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7765.375330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7765.491085] Lustre: lustre-OST0000: new disk, initializing [ 7765.492591] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7765.495703] Lustre: 343633:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7765.525778] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7767.473524] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7771.285231] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7771.643366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7771.647278] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7771.684513] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7772.734804] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7772.814815] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7774.356774] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7774.445411] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7776.738245] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7776.738625] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7776.739749] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7776.748518] Lustre: Skipped 1 previous similar message [ 7777.173726] Lustre: server umount lustre-OST0000 complete [ 7781.345244] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7781.348320] LustreError: Skipped 1 previous similar message [ 7781.350093] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7781.355711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7781.358168] Lustre: Skipped 1 previous similar message [ 7784.670306] Lustre: server umount lustre-MDT0000 complete [ 7785.993192] LustreError: 341658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789802688 with bad export cookie 11174499511041735048 [ 7785.996164] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7785.997735] LustreError: 341658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7786.117633] Lustre: server umount lustre-MDT0001 complete [ 7789.143557] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 03:24:50 (1789802690) [ 7795.078185] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 7799.005079] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 7812.626240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7812.730969] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7812.744270] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7812.784961] Lustre: lustre-MDT0000: new disk, initializing [ 7812.811079] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7812.819501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7814.364078] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7818.876681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7818.926378] Lustre: 348381:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7818.954625] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7818.958160] Lustre: Skipped 1 previous similar message [ 7818.997191] Lustre: lustre-MDT0001: new disk, initializing [ 7819.050821] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7819.074604] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7819.078806] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7820.666495] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7824.712880] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7826.634024] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7829.169321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7829.279491] Lustre: lustre-OST0000: new disk, initializing [ 7829.281519] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7829.284672] Lustre: 350351:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7829.317452] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7829.390543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7829.394159] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7829.444936] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7831.433435] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7835.583719] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7848.671399] Lustre: server umount lustre-OST0000 complete [ 7853.026927] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 7853.030309] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7853.034412] Lustre: Skipped 1 previous similar message [ 7853.036258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7853.038142] Lustre: Skipped 1 previous similar message [ 7855.073301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7855.073799] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7860.193886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7860.195822] Lustre: Skipped 2 previous similar messages [ 7864.288197] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7864.358866] Lustre: server umount lustre-MDT0000 complete [ 7865.711240] LustreError: 348375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789802768 with bad export cookie 11174499511041735916 [ 7865.713234] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7865.714738] LustreError: 348375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7876.123496] Lustre: server umount lustre-MDT0001 complete [ 7880.089394] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 7881.122761] Key type lgssc unregistered [ 7881.253369] LNet: 352503:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7881.255954] LNetError: 352503:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7881.263248] LNet: Removed LNI 192.168.204.120@tcp [ 7881.573512] Key type .llcrypt unregistered [ 7881.576657] Key type ._llcrypt unregistered [ 7914.745472] Key type ._llcrypt registered [ 7914.746481] Key type .llcrypt registered [ 7914.788226] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 7914.898341] Key type .llcrypt unregistered [ 7914.899579] Key type ._llcrypt unregistered [ 7942.519303] Key type ._llcrypt registered [ 7942.520770] Key type .llcrypt registered [ 7942.560764] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 7942.662339] Key type .llcrypt unregistered [ 7942.663441] Key type ._llcrypt unregistered [ 7954.033504] Key type ._llcrypt registered [ 7954.034560] Key type .llcrypt registered [ 7954.072587] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 7954.180402] Key type .llcrypt unregistered [ 7954.181638] Key type ._llcrypt unregistered [ 7965.640808] Key type ._llcrypt registered [ 7965.642079] Key type .llcrypt registered [ 7965.682214] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 7965.793405] Key type .llcrypt unregistered [ 7965.794503] Key type ._llcrypt unregistered [ 7993.985361] Key type ._llcrypt registered [ 7993.986368] Key type .llcrypt registered [ 7994.022776] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 7994.126444] Key type .llcrypt unregistered [ 7994.128649] Key type ._llcrypt unregistered [ 8005.778430] Key type ._llcrypt registered [ 8005.779869] Key type .llcrypt registered [ 8005.819341] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 8005.923216] Key type .llcrypt unregistered [ 8005.924512] Key type ._llcrypt unregistered [ 8017.459855] Key type ._llcrypt registered [ 8017.461270] Key type .llcrypt registered [ 8017.501433] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 8017.601249] Key type .llcrypt unregistered [ 8017.602730] Key type ._llcrypt unregistered [ 8028.917598] Key type ._llcrypt registered [ 8028.918520] Key type .llcrypt registered [ 8028.952478] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 8029.049468] Key type .llcrypt unregistered [ 8029.051218] Key type ._llcrypt unregistered [ 8040.233623] Key type ._llcrypt registered [ 8040.234738] Key type .llcrypt registered [ 8040.276500] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 8040.387267] Key type .llcrypt unregistered [ 8040.388356] Key type ._llcrypt unregistered [ 8051.377689] Key type ._llcrypt registered [ 8051.379011] Key type .llcrypt registered [ 8051.413266] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 8051.510330] Key type .llcrypt unregistered [ 8051.511417] Key type ._llcrypt unregistered [ 8072.538629] Key type ._llcrypt registered [ 8072.540717] Key type .llcrypt registered [ 8072.577174] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 8072.680431] Key type .llcrypt unregistered [ 8072.682138] Key type ._llcrypt unregistered [ 8083.590619] Key type ._llcrypt registered [ 8083.591494] Key type .llcrypt registered [ 8083.625479] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 8083.716201] Key type .llcrypt unregistered [ 8083.717213] Key type ._llcrypt unregistered [ 8094.578602] Key type ._llcrypt registered [ 8094.579695] Key type .llcrypt registered [ 8094.614338] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 8094.706380] Key type .llcrypt unregistered [ 8094.707370] Key type ._llcrypt unregistered [ 8113.758976] Key type ._llcrypt registered [ 8113.760192] Key type .llcrypt registered [ 8113.794055] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 8120.051666] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 8120.325187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8120.333336] alg: No test for adler32 (adler32-zlib) [ 8121.187278] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [ 8121.267600] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 8122.840135] Key type lgssc registered [ 8123.180684] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8134.780247] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8134.786341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8135.865267] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [ 8135.879762] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [ 8135.916874] Lustre: tst250fs-MDT0000: new disk, initializing [ 8135.936832] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8135.943363] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8137.343321] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8141.419724] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8141.459605] Lustre: 372349:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8141.488744] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [ 8141.491247] Lustre: Skipped 1 previous similar message [ 8141.529788] Lustre: tst250fs-MDT0001: new disk, initializing [ 8141.572253] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8141.584626] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8141.587682] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8143.023187] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8145.340284] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8147.566359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8147.669089] Lustre: tst250fs-OST0000: new disk, initializing [ 8147.671282] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [ 8147.674536] Lustre: 373985:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8147.707969] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8149.603395] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8151.806800] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8151.811826] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8151.852814] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8153.796517] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8153.859870] Lustre: tst250fs-OST0001: new disk, initializing [ 8153.862374] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [ 8153.866717] Lustre: 374858:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8153.903165] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8155.860142] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8158.387866] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [ 8161.650528] Lustre: 372354:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.120@tcp: IR log tst250fs-mdtir failed, not fatal: rc = -2 [ 8169.443210] Lustre: tst250fs-MDT0000-lwp-OST0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8169.444319] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 8169.446870] Lustre: Skipped 2 previous similar messages [ 8169.449314] Lustre: Skipped 1 previous similar message [ 8174.561920] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 8174.564974] Lustre: Skipped 2 previous similar messages [ 8174.637135] Lustre: server umount tst250fs-MDT0000 complete [ 8175.898751] LustreError: 372343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789803078 with bad export cookie 4410151468889411132 [ 8175.902285] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8175.903085] LustreError: 372343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8176.031692] Lustre: server umount tst250fs-MDT0001 complete [ 8186.887092] Lustre: server umount tst250fs-OST0000 complete [ 8198.434423] Lustre: server umount tst250fs-OST0001 complete [ 8204.743358] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 8207.798275] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 8218.620460] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 03:32:00 (1789803120) [ 8219.195665] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [ 8225.257754] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 8228.616080] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 8238.863404] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8006 sec ========== 03:32:20 (1789803140) [ 8239.396301] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 03:32:20 (1789803140) === [ 8240.465990] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 03:32:21 (1789803141) === [ 8253.251403] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 8254.244436] Key type lgssc unregistered [ 8254.366307] LNet: 383958:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8254.368383] LNetError: 383958:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8254.377284] LNet: Removed LNI 192.168.204.120@tcp [ 8254.689131] Key type .llcrypt unregistered [ 8254.690468] Key type ._llcrypt unregistered