[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 486701212 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002372] x2apic enabled [ 0.004004] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.008493] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.012088] LSM: Security Framework initializing [ 0.013052] Yama: becoming mindful. [ 0.014032] SELinux: Initializing. [ 0.015058] *** VALIDATE selinux *** [ 0.023586] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027764] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028146] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029114] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030127] *** VALIDATE tmpfs *** [ 0.031438] *** VALIDATE proc *** [ 0.033050] *** VALIDATE cgroup *** [ 0.034010] *** VALIDATE cgroup2 *** [ 0.035290] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036154] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038030] Spectre V2 : User space: Vulnerable [ 0.039011] Speculative Store Bypass: Vulnerable [ 0.041804] debug: unmapping init [mem 0xffffffff98c59000-0xffffffff98c60fff] [ 0.043956] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044661] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045023] ... version: 2 [ 0.045976] ... bit width: 48 [ 0.046931] ... generic registers: 4 [ 0.047014] ... value mask: 0000ffffffffffff [ 0.048016] ... max period: 00007fffffffffff [ 0.049016] ... fixed-purpose events: 3 [ 0.050014] ... event mask: 000000070000000f [ 0.051278] rcu: Hierarchical SRCU implementation. [ 0.053475] smp: Bringing up secondary CPUs ... [ 0.054589] x86: Booting SMP configuration: [ 0.055029] .... node #0, CPUs: #1 #2 #3 [ 0.058703] smp: Brought up 1 node, 4 CPUs [ 0.060013] smpboot: Max logical packages: 1 [ 0.061016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.145059] node 0 deferred pages initialised in 79ms [ 0.148316] devtmpfs: initialized [ 0.149328] x86/mm: Memory block size: 128MB [ 0.151944] gcov: version magic: 0x41383552 [ 0.152607] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.156078] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.159316] pinctrl core: initialized pinctrl subsystem [ 0.161174] [ 0.161658] ************************************************************* [ 0.164012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.166012] ** ** [ 0.168013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.170016] ** ** [ 0.173017] ** This means that this kernel is built to expose internal ** [ 0.175010] ** IOMMU data structures, which may compromise security on ** [ 0.177011] ** your system. ** [ 0.179013] ** ** [ 0.181013] ** If you see this message and you are not debugging the ** [ 0.183010] ** kernel, report this immediately to your vendor! ** [ 0.185011] ** ** [ 0.187011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.189014] ************************************************************* [ 0.191637] NET: Registered protocol family 16 [ 0.193435] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.195064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.198058] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.201032] cpuidle: using governor menu [ 0.203020] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.205598] PCI: Using configuration type 1 for base access [ 0.207121] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.216149] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.218017] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.222075] cryptd: max_cpu_qlen set to 1000 [ 0.224213] ACPI: Added _OSI(Module Device) [ 0.226016] ACPI: Added _OSI(Processor Device) [ 0.228029] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.230017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.235890] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.243087] ACPI: Interpreter enabled [ 0.246076] ACPI: PM: (supports S0 S3 S4 S5) [ 0.248013] ACPI: Using IOAPIC for interrupt routing [ 0.250123] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.254379] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.265834] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.268073] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.271033] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.274118] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.279626] acpiphp: Slot [2] registered [ 0.281145] acpiphp: Slot [5] registered [ 0.282175] acpiphp: Slot [6] registered [ 0.284181] acpiphp: Slot [7] registered [ 0.285189] acpiphp: Slot [8] registered [ 0.287146] acpiphp: Slot [9] registered [ 0.288143] acpiphp: Slot [10] registered [ 0.290134] acpiphp: Slot [3] registered [ 0.291000] acpiphp: Slot [4] registered [ 0.291161] acpiphp: Slot [11] registered [ 0.292000] acpiphp: Slot [12] registered [ 0.294115] acpiphp: Slot [13] registered [ 0.295205] acpiphp: Slot [14] registered [ 0.297099] acpiphp: Slot [15] registered [ 0.298105] acpiphp: Slot [16] registered [ 0.299241] acpiphp: Slot [17] registered [ 0.301103] acpiphp: Slot [18] registered [ 0.302116] acpiphp: Slot [19] registered [ 0.304127] acpiphp: Slot [20] registered [ 0.305103] acpiphp: Slot [21] registered [ 0.306243] acpiphp: Slot [22] registered [ 0.308106] acpiphp: Slot [23] registered [ 0.309126] acpiphp: Slot [24] registered [ 0.311107] acpiphp: Slot [25] registered [ 0.312158] acpiphp: Slot [26] registered [ 0.314111] acpiphp: Slot [27] registered [ 0.315191] acpiphp: Slot [28] registered [ 0.317127] acpiphp: Slot [29] registered [ 0.319130] acpiphp: Slot [30] registered [ 0.320117] acpiphp: Slot [31] registered [ 0.322062] PCI host bridge to bus 0000:00 [ 0.323031] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.325042] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.328042] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.330035] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.333038] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.336038] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.338260] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.341083] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.344410] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.354771] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.359858] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.362028] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.364028] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.366028] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.369356] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.371774] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.375050] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.377918] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.383016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.397020] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.405025] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.412461] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.427016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.433013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.453028] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.467287] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.473026] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.485022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.499023] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.506541] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.513021] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.518021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.536019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.547133] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.558023] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.573034] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.592019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.602511] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.608020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.616017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.637023] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.660181] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.666020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.681023] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.705021] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.715744] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.718437] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.721465] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.725461] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.728290] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.733123] iommu: Default domain type: Passthrough [ 0.735642] SCSI subsystem initialized [ 0.737202] ACPI: bus type USB registered [ 0.739106] usbcore: registered new interface driver usbfs [ 0.740101] usbcore: registered new interface driver hub [ 0.742112] usbcore: registered new device driver usb [ 0.744241] pps_core: LinuxPPS API ver. 1 registered [ 0.746015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.749086] PTP clock support registered [ 0.752055] EDAC MC: Ver: 3.0.0 [ 0.754141] PCI: Using ACPI for IRQ routing [ 0.755656] NetLabel: Initializing [ 0.757018] NetLabel: domain hash size = 128 [ 0.759014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.761079] NetLabel: unlabeled traffic allowed by default [ 0.763228] vgaarb: loaded [ 0.765264] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.766011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.771329] clocksource: Switched to clocksource kvm-clock [ 0.878846] VFS: Disk quotas dquot_6.6.0 [ 0.880581] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.883233] *** VALIDATE ramfs *** [ 0.884632] *** VALIDATE hugetlbfs *** [ 0.886293] pnp: PnP ACPI init [ 0.888648] pnp: PnP ACPI: found 6 devices [ 0.905607] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.908912] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.911119] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.913167] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.915739] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.918198] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.920874] NET: Registered protocol family 2 [ 0.923325] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.928236] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.931735] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.936720] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.939704] TCP: Hash tables configured (established 65536 bind 65536) [ 0.942424] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.945398] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.948109] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.950943] NET: Registered protocol family 1 [ 0.953902] RPC: Registered named UNIX socket transport module. [ 0.956392] RPC: Registered udp transport module. [ 0.958354] RPC: Registered tcp transport module. [ 0.960098] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.962970] NET: Registered protocol family 44 [ 0.964895] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.967351] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.969696] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.972173] PCI: CLS 0 bytes, default 64 [ 0.974569] Unpacking initramfs... [ 2.373882] debug: unmapping init [mem 0xffff8e0efcc54000-0xffff8e0efffbffff] [ 2.377529] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.380106] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.383343] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.914814] Initialise system trusted keyrings [ 2.916635] Key type blacklist registered [ 2.918646] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.931299] zbud: loaded [ 2.935889] *** VALIDATE nfs *** [ 2.937361] *** VALIDATE nfs4 *** [ 2.938909] pstore: using deflate compression [ 2.944360] Platform Keyring initialized [ 3.060745] NET: Registered protocol family 38 [ 3.062591] Key type asymmetric registered [ 3.064090] Asymmetric key parser 'x509' registered [ 3.066169] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.069430] io scheduler mq-deadline registered [ 3.070660] io scheduler kyber registered [ 3.071643] io scheduler bfq registered [ 3.073100] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.076466] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.079360] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.081970] ACPI: Power Button [PWRF] [ 3.088236] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.095976] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.114065] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.121400] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.141628] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.171667] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.210107] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.215602] Non-volatile memory driver v1.3 [ 3.217330] Linux agpgart interface v0.103 [ 3.258326] virtio_blk virtio1: [vda] 145936 512-byte logical blocks (74.7 MB/71.3 MiB) [ 3.262542] vda: detected capacity change from 0 to 74719232 [ 3.285428] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.288596] vdb: detected capacity change from 0 to 1073741824 [ 3.307229] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.310286] vdc: detected capacity change from 0 to 2621440000 [ 3.325323] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.328720] vdd: detected capacity change from 0 to 2621440000 [ 3.344198] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.347498] vde: detected capacity change from 0 to 4294967296 [ 3.363203] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.365826] vdf: detected capacity change from 0 to 4294967296 [ 3.371433] libphy: Fixed MDIO Bus: probed [ 3.376959] usbcore: registered new interface driver usbserial_generic [ 3.379957] usbserial: USB Serial support registered for generic [ 3.382605] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.385902] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.387821] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.390033] mousedev: PS/2 mouse device common for all mice [ 3.393178] rtc_cmos 00:05: RTC can wake from S4 [ 3.395101] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.396666] rtc_cmos 00:05: registered as rtc0 [ 3.399573] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.400666] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.405604] intel_pstate: CPU model not supported [ 3.407990] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.409157] hid: raw HID events driver (C) Jiri Kosina [ 3.414226] usbcore: registered new interface driver usbhid [ 3.416234] usbhid: USB HID core driver [ 3.417699] drop_monitor: Initializing network drop monitor service [ 3.419970] Initializing XFRM netlink socket [ 3.422178] NET: Registered protocol family 10 [ 3.425762] Segment Routing with IPv6 [ 3.427374] NET: Registered protocol family 17 [ 3.429584] mpls_gso: MPLS GSO support [ 3.435201] RAS: Correctable Errors collector initialized. [ 3.437203] AVX version of gcm_enc/dec engaged. [ 3.439046] AES CTR mode by8 optimization enabled [ 3.509662] sched_clock: Marking stable (3509644271, 0)->(4449562853, -939918582) [ 3.513132] registered taskstats version 1 [ 3.514982] Loading compiled-in X.509 certificates [ 3.516927] zswap: loaded using pool lzo/zbud [ 3.538222] Key type big_key registered [ 3.552334] Key type encrypted registered [ 3.553655] ima: No TPM chip found, activating TPM-bypass! [ 3.555856] ima: Allocated hash algorithm: sha1 [ 3.557649] ima: No architecture policies found [ 3.559450] evm: Initialising EVM extended attributes: [ 3.561325] evm: security.selinux [ 3.562475] evm: security.ima [ 3.563661] evm: security.capability [ 3.564785] evm: HMAC attrs: 0x1 [ 3.567143] rtc_cmos 00:05: setting system clock to 2026-07-24 23:21:45 UTC (1784935305) [ 3.573258] debug: unmapping init [mem 0xffffffff99c03000-0xffffffff99dfffff] [ 3.576696] debug: unmapping init [mem 0xffffffff98982000-0xffffffff98c58fff] [ 3.584083] Write protecting the kernel read-only data: 28672k [ 3.590117] debug: unmapping init [mem 0xffffffff97003000-0xffffffff971fffff] [ 3.593609] debug: unmapping init [mem 0xffffffff97914000-0xffffffff979fffff] [ 3.627917] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.636226] systemd[1]: Detected virtualization kvm. [ 3.638102] systemd[1]: Detected architecture x86-64. [ 3.640016] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.669135] systemd[1]: No hostname configured. [ 3.670799] systemd[1]: Set hostname to . [ 3.672243] random: systemd: uninitialized urandom read (16 bytes read) [ 3.674151] systemd[1]: Initializing machine ID from random generator. [ 3.711681] random: ln: uninitialized urandom read (6 bytes read) [ 3.803685] random: systemd: uninitialized urandom read (16 bytes read) [ 3.806870] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.812381] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.816934] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Reached target Timers. [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.364411] device-mapper: uevent: version 1.0.3 [ 4.368334] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.165332] random: fast init done [ 5.184299] virtio_net virtio0 ens2: renamed from eth0 [ 5.209232] scsi host0: ata_piix [ 5.230373] scsi host1: ata_piix [ 5.232304] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.237415] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.834636] dracut-initqueue[605]: RTNETLINK answers: File exists [ 10.069792] random: crng init done [ 10.071574] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.633842] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.853261] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.156726] SELinux: Disabled at runtime. [ 12.222477] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.230738] systemd[1]: Detected virtualization kvm. [ 12.232693] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.796067] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.799752] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.805652] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.809869] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.813558] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.823377] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.827417] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target rpc_pipefs.target. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting POSIX Message Queue File System... Mounting Kernel Debug File System... [ 12.907860] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. Starting Create list of required st…ce nodes for the current kernel... Mounting Huge Pages File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting udev Coldplug all Devices... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Apply Kernel Variables... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 13.308790] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.646772] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.648478] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.809202] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.819515] EDAC sbridge: Ver: 1.1.2 [ 15.780170] Key type dns_resolver registered [ 16.097725] NFS: Registering the id_resolver key type [ 16.101654] Key type id_resolver registered [ 16.103654] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg651-server login: [ 42.643303] libcfs: loading out-of-tree module taints kernel. [ 42.663505] Key type ._llcrypt registered [ 42.665017] Key type .llcrypt registered [ 42.718634] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 51.268510] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 52.090164] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 52.100720] alg: No test for adler32 (adler32-zlib) [ 53.140520] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [ 53.505428] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 55.143179] Key type lgssc registered [ 55.965562] Lustre: Echo OBD driver; http://www.lustre.org/ [ 74.283467] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 80.154464] hrtimer: interrupt took 3130935 ns [ 116.247771] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 128.024726] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 128.055219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 129.339116] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 129.369850] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 129.450807] Lustre: lustre-MDT0000: new disk, initializing [ 129.530331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 129.552028] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 133.855949] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 145.951369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 146.061233] Lustre: 6501:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 146.118311] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 146.125710] Lustre: Skipped 1 previous similar message [ 146.270184] Lustre: lustre-MDT0001: new disk, initializing [ 146.351732] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 146.378622] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 146.397662] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 150.794361] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 155.271555] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 164.293382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 164.767748] Lustre: lustre-OST0000: new disk, initializing [ 164.778638] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 164.785799] Lustre: 8443:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 164.842530] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 168.252451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 168.261758] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 168.334639] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 172.367448] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 186.378127] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 186.487206] Lustre: lustre-OST0001: new disk, initializing [ 186.490637] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 186.495326] Lustre: 9516:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 186.545126] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 192.040561] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 192.072705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 192.083332] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 192.158950] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 203.245850] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 208.407898] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 214.133377] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing check_logdir /tmp/testlogs/ [ 218.836804] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing yml_node [ 223.091236] Lustre: DEBUG MARKER: Client: 2.17.54.207 [ 225.401726] Lustre: DEBUG MARKER: MDS: 2.17.54.207 [ 227.459992] Lustre: DEBUG MARKER: OSS: 2.17.54.207 [ 228.887335] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Jul 24 19:25:28 EDT 2026 [ 243.873562] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 245.188696] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 263.649223] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 263.651871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 263.670086] Lustre: Skipped 1 previous similar message [ 263.678970] Lustre: Skipped 3 previous similar messages [ 266.619226] Lustre: server umount lustre-MDT0000 complete [ 269.568639] LustreError: 6493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784935571 with bad export cookie 666615720901170917 [ 269.572936] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 269.584111] LustreError: 6493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 273.888844] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 273.890329] LustreError: 6510:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 273.910226] Lustre: Skipped 2 previous similar messages [ 273.910972] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 273.939544] LustreError: 6510:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 273.961121] Lustre: Skipped 1 previous similar message [ 274.915207] LustreError: 6510:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 274.916208] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 274.930312] LustreError: 6510:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 274.944681] Lustre: Skipped 1 previous similar message [ 275.889168] Lustre: server umount lustre-MDT0001 complete [ 279.505829] Lustre: server umount lustre-OST0000 complete [ 282.786718] Lustre: server umount lustre-OST0001 complete [ 288.306560] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 296.550336] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 329.140297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 329.378476] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 329.405184] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 329.492390] Lustre: lustre-MDT0000: new disk, initializing [ 329.584881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 329.613498] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 334.241951] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 344.098949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 344.271294] Lustre: 15062:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 344.337206] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 344.342029] Lustre: Skipped 1 previous similar message [ 344.468300] Lustre: lustre-MDT0001: new disk, initializing [ 344.602455] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 344.655350] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 344.670722] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 348.551166] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 356.592392] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 362.083863] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 368.682096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 368.945648] Lustre: lustre-OST0000: new disk, initializing [ 368.949565] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 368.954115] Lustre: 17034:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 369.032550] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 370.415953] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 370.433656] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 370.558095] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 374.662644] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 382.658723] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 386.947663] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 387.153734] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 391.335665] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 391.555429] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 396.265621] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 396.269117] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 396.276606] LustreError: Skipped 1 previous similar message [ 396.291456] Lustre: Skipped 1 previous similar message [ 396.307723] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 396.318351] Lustre: Skipped 1 previous similar message [ 399.774537] Lustre: server umount lustre-OST0000 complete [ 405.992456] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 406.002723] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 406.031468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 409.728734] Lustre: server umount lustre-MDT0000 complete [ 411.616814] LustreError: 16653:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 411.637888] LustreError: 16653:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 413.214258] LustreError: 15859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784935715 with bad export cookie 666615720901172450 [ 413.217987] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 413.222296] LustreError: 15859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 413.564040] Lustre: server umount lustre-MDT0001 complete [ 418.845255] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 19:28:37 (1784935717) [ 435.409682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 435.856462] LustreError: 19908:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 435.912794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 439.968879] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 441.322297] LustreError: 19909:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 447.621592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 452.192332] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 459.152451] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 464.604086] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 466.593101] Lustre: Failing over lustre-MDT0000 [ 466.789761] Lustre: server umount lustre-MDT0000 complete [ 468.453346] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 468.464542] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 468.481217] Lustre: Skipped 1 previous similar message [ 468.490251] LustreError: 19908:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 468.502915] LustreError: 19908:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 484.219228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 484.332591] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 488.997172] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 489.958207] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 489.967783] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 489.989613] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 496.833435] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 504.292419] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 504.306334] Lustre: Skipped 1 previous similar message [ 504.308596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 504.319737] Lustre: Skipped 3 previous similar messages [ 506.814888] Lustre: server umount lustre-MDT0000 complete [ 509.412158] LustreError: 20636:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 509.452141] LustreError: 20636:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 510.618311] LustreError: 19888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784935812 with bad export cookie 666615720901173864 [ 510.624401] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 510.628498] LustreError: 19888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 511.143819] Lustre: server umount lustre-MDT0001 complete [ 518.960090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 519.369172] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 519.375173] Lustre: Skipped 2 previous similar messages [ 523.538295] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 530.362360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 534.750420] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 541.352517] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 546.112256] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 548.017192] Lustre: Failing over lustre-MDT0000 [ 548.308872] Lustre: server umount lustre-MDT0000 complete [ 549.346609] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 549.360028] LustreError: 24483:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 549.360445] Lustre: Skipped 1 previous similar message [ 549.378925] LustreError: 24483:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 565.731892] Lustre: 3636:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784935851/real 1784935851] req@ffff8e0e43778700 x1871640374986496/t0(0) o400->MGC192.168.206.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1784935867 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 565.813950] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 568.331770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 581.610375] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 581.630811] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 581.642600] Lustre: Skipped 1 previous similar message [ 581.678633] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 581.932794] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 590.779709] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 596.971447] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 596.976809] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 596.979692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 596.983995] Lustre: Skipped 1 previous similar message [ 609.247135] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 609.574572] Lustre: server umount lustre-MDT0000 complete [ 612.960987] LustreError: 23224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784935914 with bad export cookie 666615720901174802 [ 612.968680] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 612.978300] LustreError: 23224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 613.330365] Lustre: server umount lustre-MDT0001 complete [ 622.261727] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 19:32:01 (1784935921) [ 637.082468] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 647.118643] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 685.995439] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 696.389170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 696.575625] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 696.595490] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 696.691688] Lustre: lustre-MDT0000: new disk, initializing [ 696.788029] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 696.794846] Lustre: Skipped 2 previous similar messages [ 696.815202] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 701.439889] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 712.550403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 712.756403] Lustre: 29616:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 713.044597] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 717.471237] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 725.686886] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 731.198651] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 737.298794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 737.572688] Lustre: 31587:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 739.542612] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 739.593635] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 739.696561] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 739.715792] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 743.960430] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 752.681088] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 760.209325] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 780.773235] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 780.779047] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 780.791057] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 780.798166] Lustre: Skipped 1 previous similar message [ 780.813827] Lustre: Skipped 6 previous similar messages [ 783.486363] Lustre: server umount lustre-OST0000 complete [ 789.984258] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 789.992191] LustreError: Skipped 1 previous similar message [ 801.247227] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 801.534365] Lustre: server umount lustre-MDT0000 complete [ 805.158902] LustreError: 29610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784936107 with bad export cookie 666615720901175208 [ 805.159565] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 805.179076] LustreError: 29610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 805.817084] Lustre: server umount lustre-MDT0001 complete [ 816.934471] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 819.584512] Key type lgssc unregistered [ 819.878564] LNet: 33571:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 819.890779] LNetError: 33571:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 819.908095] LNet: Removed LNI 192.168.206.151@tcp [ 820.886176] Key type .llcrypt unregistered [ 820.888757] Key type ._llcrypt unregistered [ 843.577617] Key type ._llcrypt registered [ 843.578775] Key type .llcrypt registered [ 843.662676] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 844.633195] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 844.692538] alg: No test for adler32 (adler32-zlib) [ 845.742653] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [ 846.003332] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 847.703330] Key type lgssc registered [ 848.791150] Lustre: Echo OBD driver; http://www.lustre.org/ [ 855.787534] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 855.837403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 857.312079] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 862.142853] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 867.817024] LustreError: 34983:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 867.838995] LustreError: 34983:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 872.930141] LustreError: 34984:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 873.556688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 874.777915] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 880.353751] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 887.225471] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 892.157139] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 898.197694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 898.560898] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 899.575931] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 904.208747] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 911.261866] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 925.469587] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 19:37:04 (1784936224) [ 927.319508] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 929.365430] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 935.754576] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 19:37:14 (1784936234) [ 943.439404] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 19:37:22 (1784936242) [ 944.630317] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 953.811958] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 953.816269] Lustre: Skipped 2 previous similar messages [ 955.781643] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 19:37:34 (1784936254) [ 959.532839] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 959.540023] Lustre: Skipped 3 previous similar messages [ 975.355872] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 975.361540] Lustre: Skipped 6 previous similar messages [ 977.884714] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 19:37:56 (1784936276) [ 980.279510] Lustre: *** cfs_fail_loc=131b, val=2*** [ 983.427331] Lustre: *** cfs_fail_loc=131b, val=2*** [ 991.019560] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 19:38:10 (1784936290) [ 992.397501] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 992.400636] Lustre: Skipped 3 previous similar messages [ 1003.176814] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 19:38:22 (1784936302) [ 1015.266412] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1015.273528] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1015.283737] Lustre: Skipped 1 previous similar message [ 1015.294726] Lustre: Skipped 1 previous similar message [ 1020.395633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1020.398946] Lustre: Skipped 1 previous similar message [ 1021.486975] Lustre: server umount lustre-OST0000 complete [ 1025.526669] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1025.539122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1025.547735] Lustre: Skipped 1 previous similar message [ 1025.558551] Lustre: Skipped 1 previous similar message [ 1030.630077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1030.636487] Lustre: Skipped 1 previous similar message [ 1030.934907] Lustre: server umount lustre-MDT0000 complete [ 1034.655990] LustreError: 34964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784936336 with bad export cookie 5325990068145249007 [ 1034.665226] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1034.671548] LustreError: 34964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1035.032592] Lustre: server umount lustre-MDT0001 complete [ 1046.306588] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 1049.241577] Key type lgssc unregistered [ 1049.526745] LNet: 42840:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1049.540560] LNetError: 42840:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1049.558345] LNet: Removed LNI 192.168.206.151@tcp [ 1050.502103] Key type .llcrypt unregistered [ 1050.505891] Key type ._llcrypt unregistered [ 1066.764963] Key type ._llcrypt registered [ 1066.767215] Key type .llcrypt registered [ 1066.876095] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 1082.532356] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 1083.578848] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1083.599138] alg: No test for adler32 (adler32-zlib) [ 1084.689884] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [ 1084.885340] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 1086.543697] Key type lgssc registered [ 1087.793117] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1123.876675] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1123.886460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1125.190152] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1125.244822] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1125.383192] Lustre: lustre-MDT0000: new disk, initializing [ 1125.514896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1125.532605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1130.709586] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1142.030518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1142.221958] Lustre: 45992:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1142.326364] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1142.333261] Lustre: Skipped 1 previous similar message [ 1142.455240] Lustre: lustre-MDT0001: new disk, initializing [ 1142.651326] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1142.689720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1142.701735] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1147.730814] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1156.448806] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1161.863515] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1168.339421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1168.552575] Lustre: lustre-OST0000: new disk, initializing [ 1168.556177] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1168.560393] Lustre: 47962:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1168.645541] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1174.661936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1174.676346] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1174.829307] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1176.111634] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1188.360498] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1193.253768] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1193.469439] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1197.800947] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1197.969348] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1200.613289] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1200.619310] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1200.627970] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1205.759597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1205.775962] Lustre: Skipped 2 previous similar messages [ 1205.938079] Lustre: server umount lustre-OST0000 complete [ 1210.849111] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1210.860597] Lustre: Skipped 1 previous similar message [ 1210.864623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1215.667287] Lustre: server umount lustre-MDT0000 complete [ 1219.647609] LustreError: 45985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784936521 with bad export cookie 10498413759074095292 [ 1219.653409] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1219.662384] LustreError: 45985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1220.160270] Lustre: server umount lustre-MDT0001 complete [ 1229.048752] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 19:42:08 (1784936528) [ 1235.298363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1235.789332] LustreError: 50423:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1235.809524] LustreError: 50423:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1235.911631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1240.127347] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1241.061962] LustreError: 50424:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1245.165027] LustreError: 50423:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1249.041673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1249.569591] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1254.554985] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1262.582691] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1269.046932] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1275.613409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1275.969849] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1282.301470] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1290.491332] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1298.042417] LustreError: 52951:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1298.848573] Lustre: Modifying parameter general.timeout=1 in log params [ 1300.442259] Lustre: Modifying parameter general.timeout=3 in log params [ 1300.454298] Lustre: Skipped 1 previous similar message [ 1303.044391] Lustre: Modifying parameter general.timeout=6 in log params [ 1303.050415] Lustre: Skipped 2 previous similar messages [ 1307.104480] Lustre: Modifying parameter general.timeout=11 in log params [ 1307.109725] Lustre: Skipped 4 previous similar messages [ 1315.360679] Lustre: Modifying parameter general.timeout=21 in log params [ 1315.366395] Lustre: Skipped 9 previous similar messages [ 1332.248615] Lustre: Modifying parameter general.timeout=41 in log params [ 1332.252950] Lustre: Skipped 19 previous similar messages [ 1364.838520] Lustre: Modifying parameter general.timeout=80 in log params [ 1364.849590] Lustre: Skipped 38 previous similar messages [ 1417.069813] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 19:45:16 (1784936716) [ 1419.576714] Lustre: 59896:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1421.426531] Lustre: 59995:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1424.808890] Lustre: 60191:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1424.821845] Lustre: 60191:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1430.102565] Lustre: Disabling parameter general.ost.OSS.ost_io.nrs_tbf_rule= in log params [ 1430.112257] Lustre: Skipped 57 previous similar messages [ 1452.376367] Lustre: 61764:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1452.384936] Lustre: 61764:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1452.417308] LustreError: 61764:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1455.606601] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 19:45:54 (1784936754) [ 1469.921433] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1469.924396] LustreError: Skipped 1 previous similar message [ 1469.930518] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1469.941239] Lustre: Skipped 1 previous similar message [ 1469.950116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1469.952617] Lustre: Skipped 1 previous similar message [ 1473.392453] Lustre: server umount lustre-MDT0000 complete [ 1475.572389] LustreError: 50420:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1475.591266] LustreError: 50420:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1477.225372] LustreError: 59287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784936779 with bad export cookie 10498413759074096174 [ 1477.234508] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1477.243767] LustreError: 59287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1477.756122] Lustre: server umount lustre-MDT0001 complete [ 1492.231593] Lustre: server umount lustre-OST0000 complete [ 1519.308303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1519.555772] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1519.982605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1525.140939] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1533.457162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1533.648402] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1533.680979] Lustre: 64295:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1539.499845] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1548.952067] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1555.360470] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1561.345226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1561.539761] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1561.869447] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1561.879709] Lustre: Skipped 1 previous similar message [ 1567.015381] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1573.394660] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1574.922217] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1574.930339] Lustre: Skipped 15 previous similar messages [ 1574.971259] Lustre: 66864:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1581.535868] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1581.548248] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1581.562696] Lustre: Skipped 2 previous similar messages [ 1581.572103] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1581.577503] Lustre: Skipped 3 previous similar messages [ 1587.751274] Lustre: server umount lustre-OST0000 complete [ 1592.289042] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1592.296326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1592.314192] Lustre: Skipped 2 previous similar messages [ 1592.317444] Lustre: Skipped 4 previous similar messages [ 1597.659555] Lustre: server umount lustre-MDT0000 complete [ 1601.016585] LustreError: 64289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784936902 with bad export cookie 10498413759074099184 [ 1601.023316] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1601.038488] LustreError: 64289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1601.515734] Lustre: server umount lustre-MDT0001 complete [ 1614.156888] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 1617.524342] Key type lgssc unregistered [ 1617.911527] LNet: 68076:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1617.919206] LNetError: 68076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1617.943276] LNet: Removed LNI 192.168.206.151@tcp [ 1619.036761] Key type .llcrypt unregistered [ 1619.038730] Key type ._llcrypt unregistered [ 1643.512279] Key type ._llcrypt registered [ 1643.514296] Key type .llcrypt registered [ 1643.621923] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 1644.643386] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1644.661738] alg: No test for adler32 (adler32-zlib) [ 1645.709923] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [ 1645.975111] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 1647.607279] Key type lgssc registered [ 1648.733141] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1656.454113] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1656.481724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1658.039327] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1662.266972] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1667.562934] LustreError: 69489:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1667.580747] LustreError: 69489:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1669.253470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1669.803645] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1673.678285] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1680.197500] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1684.846781] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1690.306973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1690.876816] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1695.651047] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1701.718514] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1709.814496] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1720.800700] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1720.804514] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1720.810490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1722.850459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1722.865571] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1722.882547] Lustre: Skipped 1 previous similar message [ 1725.018933] Lustre: server umount lustre-MDT0000 complete [ 1727.970902] LustreError: 69490:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1727.979844] LustreError: 69490:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1728.841361] LustreError: 69472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784937030 with bad export cookie 16988345265745312514 [ 1728.844444] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1728.874308] LustreError: 69472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1729.251599] Lustre: server umount lustre-MDT0001 complete [ 1742.379589] Lustre: server umount lustre-OST0000 complete [ 1767.837247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1768.173853] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1768.231375] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1768.571552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1773.394408] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1781.229277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1781.360233] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1781.390423] Lustre: 74360:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1781.806277] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1785.514311] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1791.462221] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1796.138343] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1802.231174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1802.423216] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1802.779242] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1809.223172] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1816.655665] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1818.390535] Lustre: 76934:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1820.436817] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1830.370339] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1830.384666] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1830.426251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1830.439073] Lustre: Skipped 2 previous similar messages [ 1831.957742] Lustre: server umount lustre-OST0000 complete [ 1838.048650] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1838.063842] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1838.093631] Lustre: Skipped 1 previous similar message [ 1838.115827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1838.121272] Lustre: Skipped 1 previous similar message [ 1842.784514] Lustre: server umount lustre-MDT0000 complete [ 1846.240965] LustreError: 74370:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1846.264238] LustreError: 74370:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1846.333761] LustreError: 76937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784937148 with bad export cookie 16988345265745313347 [ 1846.336955] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1846.339655] LustreError: 76937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1846.961696] Lustre: server umount lustre-MDT0001 complete [ 1857.207690] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 1860.013775] Key type lgssc unregistered [ 1860.257221] LNet: 78149:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1860.269623] LNetError: 78149:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1860.283466] LNet: Removed LNI 192.168.206.151@tcp [ 1861.292685] Key type .llcrypt unregistered [ 1861.299956] Key type ._llcrypt unregistered [ 1885.859900] Key type ._llcrypt registered [ 1885.862073] Key type .llcrypt registered [ 1885.951226] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 1886.733575] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1886.867862] alg: No test for adler32 (adler32-zlib) [ 1887.995830] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [ 1888.198070] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 1889.847169] Key type lgssc registered [ 1890.787726] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1898.597825] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1898.615285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1900.137727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1905.537537] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1910.761364] LustreError: 79565:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1910.777984] LustreError: 79565:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1914.567491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1914.955476] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1919.537615] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1926.414409] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1931.319960] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1937.328464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1938.022386] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1944.820699] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1952.817528] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1960.414978] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1961.646368] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1963.780557] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1963.800579] Lustre: Skipped 1 previous similar message [ 1966.213668] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1966.219261] Lustre: Skipped 1 previous similar message [ 1971.005565] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1971.011724] Lustre: Skipped 4 previous similar messages [ 1979.099221] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1979.103368] Lustre: Skipped 9 previous similar messages [ 1995.899723] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1995.903713] Lustre: Skipped 19 previous similar messages [ 2027.936062] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2027.946797] Lustre: Skipped 34 previous similar messages [ 2057.333972] Lustre: 87666:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2060.426332] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 19:55:59 (1784937359) [ 2062.113570] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2064.492390] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 19:56:03 (1784937363) [ 2069.473694] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2069.482784] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2069.493909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2073.569408] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2073.576768] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2073.598795] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2073.614026] Lustre: Skipped 1 previous similar message [ 2075.794326] Lustre: server umount lustre-OST0000 complete [ 2083.809040] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2083.811588] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2083.827314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2083.835744] Lustre: Skipped 1 previous similar message [ 2086.601628] Lustre: server umount lustre-MDT0000 complete [ 2088.934076] LustreError: 79584:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2088.949524] LustreError: 79584:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2090.498979] LustreError: 82151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784937392 with bad export cookie 9229260662383290860 [ 2090.502346] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2090.510379] LustreError: 82151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2090.842909] Lustre: server umount lustre-MDT0001 complete [ 2102.944103] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 2106.138664] Key type lgssc unregistered [ 2106.440712] LNet: 88915:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2106.455841] LNetError: 88915:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2106.473985] LNet: Removed LNI 192.168.206.151@tcp [ 2107.313612] Key type .llcrypt unregistered [ 2107.315246] Key type ._llcrypt unregistered [ 2111.859040] Key type ._llcrypt registered [ 2111.862206] Key type .llcrypt registered [ 2112.024386] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2116.162466] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules [ 2117.274541] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2117.305095] alg: No test for adler32 (adler32-zlib) [ 2118.609047] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [ 2118.879541] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 2120.535158] Key type lgssc registered [ 2121.713167] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2123.084632] LustreError: 90117:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2139.846323] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 2144.840299] LustreError: 90117:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2148.976964] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2149.006074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2150.660052] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2155.322963] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2160.098826] LustreError: 91140:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2160.150093] LustreError: 91140:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2163.726140] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 19:57:42 (1784937462) [ 2165.223063] LustreError: 91141:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2168.179032] LustreError: 91173:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2168.199574] LustreError: 91173:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2168.211620] LustreError: 91173:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 2168.568340] Lustre: server umount lustre-MDT0000 complete [ 2181.739714] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 2184.692889] Key type lgssc unregistered [ 2185.011578] LNet: 92667:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2185.024685] LNetError: 92667:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2185.039696] LNet: Removed LNI 192.168.206.151@tcp [ 2185.750733] Key type .llcrypt unregistered [ 2185.752745] Key type ._llcrypt unregistered [ 2205.173562] Key type ._llcrypt registered [ 2205.181508] Key type .llcrypt registered [ 2205.315758] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 2206.813597] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2206.844827] alg: No test for adler32 (adler32-zlib) [ 2207.958821] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [ 2208.247356] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 2209.951158] Key type lgssc registered [ 2211.302201] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2218.063962] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2218.103143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2219.794945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2223.855659] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2229.220374] LustreError: 93887:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2229.237988] LustreError: 93887:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2232.269815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2232.974341] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2237.801549] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2245.271952] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2251.178230] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2257.986336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2258.691070] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2264.751894] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2271.361832] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2323.284701] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 20:00:22 (1784937622) [ 2324.847852] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2326.758138] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 20:00:25 (1784937625) [ 2333.667503] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2333.676141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2333.692082] Lustre: Skipped 1 previous similar message [ 2333.710572] Lustre: Skipped 2 previous similar messages [ 2337.051952] Lustre: server umount lustre-MDT0000 complete [ 2338.786467] LustreError: 93886:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2340.439261] LustreError: 93867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784937642 with bad export cookie 3471961653228677289 [ 2340.446247] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2340.458633] LustreError: 93867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2340.785627] Lustre: server umount lustre-MDT0001 complete [ 2354.815519] Lustre: server umount lustre-OST0000 complete [ 2366.528857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2366.901293] LustreError: 98194:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2366.922868] LustreError: 98194:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2367.108835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2371.751265] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2372.580936] LustreError: 98195:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2372.616906] LustreError: 98195:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2376.674072] LustreError: 98194:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2379.905919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2380.690708] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2384.929462] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2392.653310] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2398.916547] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2412.789273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2413.019476] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2413.035212] LustreError: 98181:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2413.077862] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2413.088692] LustreError: 100279:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2413.096219] LustreError: 100279:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2413.100029] LustreError: 100279:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2413.173066] Lustre: server umount lustre-OST0000 complete [ 2413.177287] LustreError: 100279:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2421.236991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2421.446237] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2421.452468] LustreError: 98182:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2421.471072] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2421.479939] LustreError: 100740:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2421.494222] LustreError: 100740:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2421.502699] LustreError: 100740:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2421.588130] Lustre: server umount lustre-OST0000 complete [ 2421.596214] LustreError: 100740:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2430.982933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2431.185958] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2431.196685] Lustre: Found index 0 for lustre-OST0000, updating log [ 2431.206811] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2431.302368] Lustre: lustre-OST0000: new disk, initializing [ 2431.307538] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2431.319788] Lustre: 101276:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2431.661732] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2439.181602] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2443.106678] Lustre: Failing over lustre-OST0000 [ 2443.414611] Lustre: server umount lustre-OST0000 complete [ 2450.849218] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2450.853462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2450.877505] Lustre: Skipped 2 previous similar messages [ 2450.896635] Lustre: Skipped 1 previous similar message [ 2456.033737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2456.048759] Lustre: Skipped 1 previous similar message [ 2461.157349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2461.169329] Lustre: Skipped 1 previous similar message [ 2462.178416] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2462.302700] Lustre: server umount lustre-MDT0000 complete [ 2466.279171] LustreError: 99440:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2466.307578] LustreError: 99440:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2467.089858] LustreError: 98175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784937769 with bad export cookie 3471961653228680040 [ 2467.093982] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2467.130899] LustreError: 98175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2467.802824] Lustre: server umount lustre-MDT0001 complete [ 2476.648335] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 20:02:55 (1784937775) [ 2492.422420] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 2501.930358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2502.396149] LustreError: 103622:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2502.510615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2507.349549] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2516.099306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2520.617289] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2523.064145] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2528.597855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2529.306706] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2534.536929] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2541.538051] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2541.543659] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2541.555351] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2541.977859] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2542.021576] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2542.083768] Lustre: lustre-OST0001: new disk, initializing [ 2542.087503] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2542.094379] Lustre: 105942:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2542.361399] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2542.391409] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2542.404113] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2542.416214] Lustre: Skipped 2 previous similar messages [ 2547.554957] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:03 [ 2547.567490] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2547.574449] Lustre: lustre-OST0000: Recovery over after 0:06, of 1 clients 1 recovered and 0 were evicted. [ 2547.582448] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2547.593150] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2548.716453] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2551.450219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2551.463518] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2551.592809] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2558.756629] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2566.957331] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2571.579483] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.51@tcp (stopping) [ 2572.263596] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2577.118659] Lustre: server umount lustre-MDT0001 complete [ 2577.379465] LustreError: 103622:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2577.400953] LustreError: 103622:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2587.821982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2587.890074] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2587.895184] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2587.947904] Lustre: 103611:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2589.275036] LustreError: 107471:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2589.292376] LustreError: 107471:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2589.314418] LustreError: 103614:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2589.462849] LustreError: 107484:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.206.151@tcp: cfg command failed: rc = -17 [ 2589.471138] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2589.471138] [ 2594.298234] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2594.353758] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2603.635435] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 20:05:02 (1784937902) [ 2619.288841] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 2634.409609] Lustre: 109555:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2654.170794] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2657.499799] Lustre: 110690:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2746.853056] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2746.864044] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2746.871910] Lustre: Skipped 2 previous similar messages [ 2746.875039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2746.879133] Lustre: Skipped 4 previous similar messages [ 2752.313191] Lustre: server umount lustre-MDT0000 complete [ 2752.992956] LustreError: 105762:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2753.008837] LustreError: 105762:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 2755.626677] LustreError: 103603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784938057 with bad export cookie 3471961653228680971 [ 2755.629373] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2755.637473] LustreError: 103603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2755.846241] Lustre: server umount lustre-MDT0001 complete [ 2770.307725] Lustre: server umount lustre-OST0000 complete [ 2784.555563] Lustre: server umount lustre-OST0001 complete [ 2790.218161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2799.315249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2808.271370] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2819.084292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2837.154376] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 2846.591877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2846.611374] Lustre: lustre-MDT0000: reset Object Index mappings [ 2847.208325] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2847.211883] Lustre: Skipped 1 previous similar message [ 2852.130487] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2860.801106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2860.859428] Lustre: lustre-MDT0001: reset Object Index mappings [ 2866.652661] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2870.039550] Lustre: 115792:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2877.053134] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2883.964187] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2887.085418] LustreError: 116146:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2887.105338] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:199 to 0x280000400:225) [ 2887.111630] LustreError: 116146:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2892.218490] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:382 to 0x280000401:418) [ 2892.459899] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2898.440510] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 2898.456471] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 2899.734485] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2906.650644] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2910.302116] Lustre: 117668:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2958.462217] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 20:10:57 (1784938257) [ 2985.440577] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2985.455191] Lustre: Skipped 6 previous similar messages [ 2985.461877] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2985.466157] Lustre: Skipped 7 previous similar messages [ 2991.844577] Lustre: server umount lustre-MDT0000 complete [ 2995.999443] LustreError: 114626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784938297 with bad export cookie 3471961653228728046 [ 2996.014071] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2996.020788] LustreError: 114626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3000.805219] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3002.538331] Lustre: server umount lustre-MDT0001 complete [ 3012.372315] Lustre: server umount lustre-OST0000 complete [ 3023.108776] Lustre: server umount lustre-OST0001 complete [ 3030.389317] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 3040.893413] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 3074.879854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3075.160699] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3075.191956] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3075.254991] Lustre: lustre-MDT0000: new disk, initializing [ 3075.320612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3075.329435] Lustre: Skipped 3 previous similar messages [ 3075.350600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3079.803928] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3086.067365] Lustre: server umount lustre-MDT0000 complete [ 3094.605765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3094.909976] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3094.953828] Lustre: 121938:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3094.979092] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3099.730991] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3104.497647] Lustre: server umount lustre-MDT0000 complete [ 3112.553653] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 20:13:31 (1784938411) [ 3113.827809] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3115.489831] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 20:13:34 (1784938414) [ 3133.416399] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 3145.023514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3150.914541] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3159.950783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3160.074819] Lustre: 123988:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3160.162102] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3160.166496] Lustre: Skipped 1 previous similar message [ 3160.302842] Lustre: lustre-MDT0001: new disk, initializing [ 3160.440702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3160.458777] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3165.203083] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3170.209837] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3176.097301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3176.504466] Lustre: lustre-OST0000: new disk, initializing [ 3176.514595] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3176.520435] Lustre: 125566:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3178.043076] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3178.058492] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3178.120299] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3183.048667] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3193.589913] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3193.799932] Lustre: lustre-OST0001: new disk, initializing [ 3193.803439] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3193.807720] Lustre: 126436:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3195.444115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3195.462908] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3195.573734] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3200.319985] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3210.383548] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3214.165696] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3589.377570] Lustre: 132849:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3589.384406] Lustre: 132849:0:(osd_handler.c:2091:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3589.389621] Lustre: 132849:0:(osd_handler.c:2098:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3589.397276] Lustre: 132849:0:(osd_handler.c:2108:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3589.404943] Lustre: 132849:0:(osd_handler.c:2115:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3589.409536] Lustre: 132849:0:(osd_handler.c:2122:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3855.864769] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 20:25:54 (1784939154) [ 3871.717746] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3871.725212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3871.733127] Lustre: Skipped 2 previous similar messages [ 3871.749147] Lustre: Skipped 11 previous similar messages [ 3876.836656] LustreError: 127240:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3876.861570] LustreError: 127240:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 3878.627909] Lustre: server umount lustre-MDT0000 complete [ 3882.378153] LustreError: 124739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784939184 with bad export cookie 3471961653228794714 [ 3882.383541] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3882.392711] LustreError: 124739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3887.073037] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3887.100241] Lustre: Skipped 2 previous similar messages [ 3887.112232] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3887.118840] Lustre: Skipped 1 previous similar message [ 3888.862930] Lustre: server umount lustre-MDT0001 complete [ 3898.833170] Lustre: server umount lustre-OST0000 complete [ 3908.506318] Lustre: server umount lustre-OST0001 complete [ 3914.681048] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 3924.112289] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 3958.880848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3959.159276] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3959.182822] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3959.274760] Lustre: lustre-MDT0000: new disk, initializing [ 3959.344865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3959.349728] Lustre: Skipped 5 previous similar messages [ 3959.360503] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3964.045661] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3975.086779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3975.187869] Lustre: 135899:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3975.239757] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3975.243616] Lustre: Skipped 1 previous similar message [ 3975.332058] Lustre: lustre-MDT0001: new disk, initializing [ 3975.515351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3975.525386] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3980.474134] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3989.917456] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3995.496547] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4002.302139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4002.649336] Lustre: lustre-OST0000: new disk, initializing [ 4002.654123] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4002.667369] Lustre: 137867:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4002.806600] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4002.816591] Lustre: Skipped 1 previous similar message [ 4004.356364] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4004.378168] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4004.495295] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4009.275467] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4018.183374] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4027.634879] Lustre: lustre-MDD0000: changelog on [ 4030.554538] Lustre: lustre-MDD0001: changelog on [ 5725.715722] Lustre: lustre-MDD0001: changelog off [ 5727.372689] Lustre: 135905:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 5728.244995] Lustre: lustre-MDD0000: changelog off [ 5734.882056] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5734.884089] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5734.898619] LustreError: Skipped 1 previous similar message [ 5734.900375] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5734.905070] Lustre: Skipped 2 previous similar messages [ 5737.965459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5737.974766] Lustre: Skipped 2 previous similar messages [ 5743.077491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5743.084750] Lustre: Skipped 1 previous similar message [ 5747.167148] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5747.302848] Lustre: server umount lustre-OST0000 complete [ 5752.288503] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5752.302797] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5752.337685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5753.331180] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5757.606708] Lustre: server umount lustre-MDT0000 complete [ 5758.436402] LustreError: 135909:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5758.455287] LustreError: 135909:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 5760.798467] LustreError: 135891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784941062 with bad export cookie 3471961653229614666 [ 5760.799470] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5760.811645] LustreError: 135891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5761.094516] Lustre: server umount lustre-MDT0001 complete [ 5770.973799] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 5773.585614] Key type lgssc unregistered [ 5773.885704] LNet: 146974:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5773.898297] LNetError: 146974:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5773.918170] LNet: Removed LNI 192.168.206.151@tcp [ 5774.898119] Key type .llcrypt unregistered [ 5774.900107] Key type ._llcrypt unregistered [ 5794.159339] Key type ._llcrypt registered [ 5794.161502] Key type .llcrypt registered [ 5794.253646] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 5810.120840] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 5811.219686] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5811.269670] alg: No test for adler32 (adler32-zlib) [ 5812.383669] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [ 5812.654300] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 5814.351528] Key type lgssc registered [ 5815.425612] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5848.811597] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5848.847691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5850.069649] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5850.122354] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5850.241667] Lustre: lustre-MDT0000: new disk, initializing [ 5850.326783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5850.362222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5855.627833] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5865.907694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5866.029897] Lustre: 150175:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5866.088638] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5866.094283] Lustre: Skipped 1 previous similar message [ 5866.183764] Lustre: lustre-MDT0001: new disk, initializing [ 5866.300906] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5866.394962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5866.415614] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5870.433415] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5880.805831] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5886.858602] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5894.537030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5894.857952] Lustre: lustre-OST0000: new disk, initializing [ 5894.862609] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5894.868185] Lustre: 152147:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5894.991463] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5901.056764] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5902.377325] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5902.389916] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5902.566718] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5911.410623] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5918.183899] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_module obdecho/obdecho [ 5920.052994] LustreError: 153151:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 5920.061856] LustreError: 153151:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 5921.637057] LustreError: 153248:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 5933.025035] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5933.035292] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5933.048689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5933.546466] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5933.559510] Lustre: Skipped 1 previous similar message [ 5933.569299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5933.571587] Lustre: Skipped 1 previous similar message [ 5938.347521] Lustre: server umount lustre-MDT0000 complete [ 5941.415320] LustreError: 150167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784941243 with bad export cookie 5394080904746319666 [ 5941.423354] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5941.428667] LustreError: 150167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5941.698578] Lustre: server umount lustre-MDT0001 complete [ 5955.609868] Lustre: server umount lustre-OST0000 complete [ 5974.449549] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 5984.196200] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 6020.401437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6020.689548] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6020.728295] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6020.858297] Lustre: lustre-MDT0000: new disk, initializing [ 6020.994540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6021.028393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6025.336245] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6034.674469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6034.774173] Lustre: 156972:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6034.862287] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6034.867820] Lustre: Skipped 1 previous similar message [ 6034.936285] Lustre: lustre-MDT0001: new disk, initializing [ 6034.997900] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6035.041869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6035.050546] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6038.911957] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6046.617689] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6050.787931] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6055.942450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6056.155461] Lustre: lustre-OST0000: new disk, initializing [ 6056.159319] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6056.167973] Lustre: 158942:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6056.238920] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6061.389536] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6063.139408] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6063.148826] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6063.239478] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6069.009187] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6072.500185] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6072.703907] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6076.918951] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6077.129115] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6078.944150] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6078.954049] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6078.971856] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6083.553458] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6083.555221] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6088.694537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6088.702699] Lustre: Skipped 2 previous similar messages [ 6093.279567] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6093.597634] Lustre: server umount lustre-OST0000 complete [ 6098.916140] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6098.924541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6098.932241] Lustre: Skipped 1 previous similar message [ 6103.414835] Lustre: server umount lustre-MDT0000 complete [ 6104.032777] LustreError: 158136:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6104.050482] LustreError: 158136:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6106.697147] LustreError: 156966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784941408 with bad export cookie 5394080904746320625 [ 6106.699830] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6106.719034] LustreError: 156966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6107.249855] Lustre: server umount lustre-MDT0001 complete [ 6110.237478] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 21:03:29 (1784941409) [ 6111.668889] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6113.444073] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 21:03:32 (1784941412) [ 6120.997788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6121.578196] LustreError: 161353:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6121.649684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6126.370729] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6127.086733] LustreError: 161354:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6131.178187] LustreError: 161353:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6135.092673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6139.955466] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6147.703494] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6153.775953] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6160.321831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6160.692563] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6160.697834] Lustre: Skipped 1 previous similar message [ 6165.826738] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6171.650630] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6187.818500] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.51@tcp (stopping) [ 6187.825914] Lustre: Skipped 1 previous similar message [ 6190.568331] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6190.583172] Lustre: Skipped 2 previous similar messages [ 6193.952786] Lustre: server umount lustre-MDT0000 complete [ 6195.682903] LustreError: 162788:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6195.692187] LustreError: 162788:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6197.014956] LustreError: 161334:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784941498 with bad export cookie 5394080904746321507 [ 6197.025361] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6203.350633] Lustre: server umount lustre-MDT0001 complete [ 6208.520294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6218.909149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6219.268314] LustreError: 165217:0:(llog.c:1655:llog_backup()) MGC192.168.206.151@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6219.277169] Lustre: 165217:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.151@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6226.402225] LustreError: 165222:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6226.416575] LustreError: 165222:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 6244.114711] LustreError: 165224:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6244.136064] LustreError: 165224:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 6277.088084] LustreError: 165223:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6277.104044] LustreError: 165223:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 24 previous similar messages [ 6280.671467] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6280.675330] Lustre: 165217:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.206.151@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6321.662847] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6322.734242] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6326.060766] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6333.285508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6341.090498] LustreError: 163185:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6341.093325] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x4adba000f91f5fbc [ 6341.116594] LustreError: 163185:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 46 previous similar messages [ 6341.137927] Lustre: MGC192.168.206.151@tcp: Connection restored to 0@lo (at 0@lo) [ 6345.279968] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6346.739319] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6351.619703] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6352.868406] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6357.758333] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6359.253047] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6376.572809] Lustre: server umount lustre-MDT0000 complete [ 6377.443630] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6377.444583] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6377.457398] Lustre: Skipped 3 previous similar messages [ 6379.478539] LustreError: 161333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784941681 with bad export cookie 5394080904746327996 [ 6379.482969] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6379.493260] LustreError: 161333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 6379.813077] Lustre: server umount lustre-MDT0001 complete [ 6392.273848] Lustre: server umount lustre-OST0000 complete [ 6410.549996] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 6419.077626] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 6449.725232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6450.004267] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6450.053330] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6450.169443] Lustre: lustre-MDT0000: new disk, initializing [ 6450.246343] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6450.251128] Lustre: Skipped 1 previous similar message [ 6450.276476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6453.839875] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6462.204039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6462.269262] Lustre: 170319:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6462.523149] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6466.052951] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6473.231073] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6477.311285] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6481.994371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6482.221286] Lustre: lustre-OST0000: new disk, initializing [ 6482.223654] Lustre: Skipped 1 previous similar message [ 6482.227507] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6482.230560] Lustre: Skipped 2 previous similar messages [ 6482.235241] Lustre: 172288:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6486.984612] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6491.686326] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6491.696306] Lustre: Skipped 1 previous similar message [ 6491.707407] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6491.753980] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6494.760389] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6498.794902] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6498.954510] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6502.573910] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6502.718252] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6506.979372] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6506.982209] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6506.994526] Lustre: Skipped 1 previous similar message [ 6507.011351] Lustre: Skipped 9 previous similar messages [ 6510.546140] Lustre: server umount lustre-OST0000 complete [ 6513.632220] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6519.942630] Lustre: server umount lustre-MDT0000 complete [ 6522.336837] LustreError: 170326:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6522.346733] LustreError: 170326:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 6523.232378] LustreError: 170314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784941825 with bad export cookie 5394080904746344796 [ 6523.241045] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6523.243699] LustreError: 170314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6523.412036] Lustre: server umount lustre-MDT0001 complete [ 6530.045050] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 21:10:29 (1784941829) [ 6534.931897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6538.839695] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6545.869207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6550.149812] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6556.814821] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6561.491211] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6566.320738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6570.925363] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6576.128320] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6588.949716] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 21:11:28 (1784941888) [ 6592.480583] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6592.486085] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6592.493809] Lustre: Skipped 2 previous similar messages [ 6592.500721] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6592.504918] Lustre: Skipped 3 previous similar messages [ 6598.275922] Lustre: server umount lustre-OST0000 complete [ 6607.422941] Lustre: server umount lustre-MDT0000 complete [ 6610.350733] LustreError: 174677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784941912 with bad export cookie 5394080904746345671 [ 6610.363605] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6610.555440] Lustre: server umount lustre-MDT0001 complete [ 6619.960821] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 6622.326416] Key type lgssc unregistered [ 6622.593788] LNet: 178422:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6622.602301] LNetError: 178422:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6622.615164] LNet: Removed LNI 192.168.206.151@tcp [ 6623.441428] Key type .llcrypt unregistered [ 6623.448963] Key type ._llcrypt unregistered [ 6638.704897] Key type ._llcrypt registered [ 6638.708433] Key type .llcrypt registered [ 6638.827166] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 6639.826467] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6639.836897] alg: No test for adler32 (adler32-zlib) [ 6640.874370] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [ 6641.109060] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 6642.807170] Key type lgssc registered [ 6643.701637] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6649.703535] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6649.736439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6665.249490] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6670.367471] LustreError: 179669:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.151@tcp: failed processing log, type 4: rc = -110 [ 6675.487653] LustreError: 179669:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6675.506204] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 6675.512437] LustreError: MGC192.168.206.151@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 6675.522536] LustreError: 179669:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 6675.529992] LustreError: 179669:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 6675.537268] LustreError: 179669:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 6675.581220] Lustre: server umount lustre-OST0000 complete [ 6675.584325] LustreError: 179669:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6682.117700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6683.421141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6686.601965] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6692.558506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6692.840263] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6696.297558] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6701.374332] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6705.064843] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6709.833611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6709.970665] LustreError: 181984:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6709.978677] Lustre: 181984:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.206.151@tcp: invalid local config log lustre-OST0000: rc = -22 [ 6710.069625] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6714.344476] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6719.657954] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6722.530726] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6722.546629] Lustre: Skipped 1 previous similar message [ 6722.548102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6722.556609] Lustre: Skipped 2 previous similar messages [ 6727.354166] Lustre: server umount lustre-MDT0000 complete [ 6727.650932] LustreError: 180170:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6727.680385] LustreError: 180170:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 6729.963633] LustreError: 180150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784942031 with bad export cookie 7809884203644745857 [ 6729.967973] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6729.974228] LustreError: 180150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6730.262355] Lustre: server umount lustre-MDT0001 complete [ 6743.470233] Lustre: server umount lustre-OST0000 complete [ 6750.192672] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 6757.306974] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 6783.337632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6783.521455] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6783.564787] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6783.618199] Lustre: lustre-MDT0000: new disk, initializing [ 6783.690120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6783.707680] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6787.175203] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6794.640432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6794.751717] Lustre: 185504:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6794.822426] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6794.826238] Lustre: Skipped 1 previous similar message [ 6794.907191] Lustre: lustre-MDT0001: new disk, initializing [ 6794.991978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6795.022733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6795.030087] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6798.750160] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6806.224656] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6810.502833] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6815.930742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6816.172682] Lustre: lustre-OST0000: new disk, initializing [ 6816.175628] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6816.181772] Lustre: 187471:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6816.255363] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6821.036694] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6826.013469] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6826.019279] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6826.117669] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6828.333588] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6831.831861] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6831.992640] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6835.334892] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6835.469953] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6841.318566] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6841.320406] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6841.327692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6841.337971] Lustre: Skipped 2 previous similar messages [ 6841.351700] Lustre: Skipped 1 previous similar message [ 6846.433206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6846.438613] Lustre: Skipped 1 previous similar message [ 6851.039412] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6851.200604] Lustre: server umount lustre-OST0000 complete [ 6856.674952] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6856.676916] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6856.683292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6856.695528] Lustre: Skipped 1 previous similar message [ 6856.706865] Lustre: Skipped 1 previous similar message [ 6860.209280] Lustre: server umount lustre-MDT0000 complete [ 6861.794434] LustreError: 185513:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6861.812672] LustreError: 185513:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6862.986628] LustreError: 185498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784942164 with bad export cookie 7809884203644746571 [ 6862.991968] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6862.997184] LustreError: 185498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6863.283983] Lustre: server umount lustre-MDT0001 complete [ 6869.690280] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 21:16:09 (1784942169) [ 6882.762191] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 6885.063630] Key type lgssc unregistered [ 6885.307714] LNet: 190416:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6885.311969] LNetError: 190416:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6885.323786] LNet: Removed LNI 192.168.206.151@tcp [ 6886.078855] Key type .llcrypt unregistered [ 6886.081349] Key type ._llcrypt unregistered [ 6899.380396] Key type ._llcrypt registered [ 6899.382506] Key type .llcrypt registered [ 6899.471155] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 6900.328130] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6900.352491] alg: No test for adler32 (adler32-zlib) [ 6901.406267] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [ 6901.639764] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 6903.303544] Key type lgssc registered [ 6904.183528] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6910.258247] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6910.288333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6911.558318] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6915.339563] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6921.709729] LustreError: 191614:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6921.729972] LustreError: 191614:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6922.029636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6922.376494] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6925.846075] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6931.562704] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6935.933693] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6941.979657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6942.154605] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6942.157492] LustreError: 193476:0:(llog.c:1696:llog_backup()) MGC192.168.206.151@tcp: failed to backup log lustre-OST0000: rc = -28 [ 6942.169379] Lustre: 193476:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.206.151@tcp: can't backup local config lustre-OST0000: rc = -28 [ 6942.252349] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6947.118296] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6952.387857] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6957.541168] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6957.547085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6957.555542] Lustre: Skipped 1 previous similar message [ 6962.658539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6962.670440] Lustre: Skipped 4 previous similar messages [ 6967.780139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6967.789928] Lustre: Skipped 2 previous similar messages [ 6968.287798] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6968.462436] Lustre: server umount lustre-MDT0000 complete [ 6970.943255] LustreError: 191595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784942272 with bad export cookie 17976360607169225067 [ 6970.945032] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6970.950885] LustreError: 191595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6971.193082] Lustre: server umount lustre-MDT0001 complete [ 6984.628170] Lustre: server umount lustre-OST0000 complete [ 6992.709217] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 7001.246578] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 7027.737884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7027.931525] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7027.950410] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7028.008775] Lustre: lustre-MDT0000: new disk, initializing [ 7028.079831] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7028.097871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7031.062752] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7038.393268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7038.475941] Lustre: 197008:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7038.505620] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7038.508406] Lustre: Skipped 1 previous similar message [ 7038.511984] LustreError: 197788:0:(llog.c:1696:llog_backup()) MGC192.168.206.151@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7038.521662] LustreError: 197788:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7038.528906] Lustre: 197788:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.151@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7038.557530] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7038.561820] Lustre: Skipped 1 previous similar message [ 7038.617156] Lustre: lustre-MDT0001: new disk, initializing [ 7038.687694] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7038.708282] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7038.714907] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7041.747699] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7048.511751] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7052.370607] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7057.124981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7057.283926] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7057.285645] Lustre: Skipped 1 previous similar message [ 7057.287758] LustreError: 198948:0:(llog.c:1696:llog_backup()) MGC192.168.206.151@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7057.292678] LustreError: 198948:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7057.300694] Lustre: 198948:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.151@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7057.306393] Lustre: 198948:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7057.319514] Lustre: lustre-OST0000: new disk, initializing [ 7057.324641] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7057.328972] Lustre: 198971:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7057.391410] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7061.891823] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7064.625386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7064.635027] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7064.718109] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7068.694451] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7071.630618] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7071.777506] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7075.198702] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7075.364641] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7080.419110] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7080.426593] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7080.440423] Lustre: Skipped 1 previous similar message [ 7080.444048] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7080.447498] Lustre: Skipped 1 previous similar message [ 7085.551286] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7085.556171] Lustre: Skipped 1 previous similar message [ 7091.169157] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7091.287247] Lustre: server umount lustre-OST0000 complete [ 7095.267712] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7095.283539] LustreError: Skipped 1 previous similar message [ 7095.289620] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7095.304562] Lustre: Skipped 1 previous similar message [ 7095.309532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7095.312037] Lustre: Skipped 2 previous similar messages [ 7100.208327] Lustre: server umount lustre-MDT0000 complete [ 7100.901028] LustreError: 197015:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7100.914927] LustreError: 197015:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7102.908100] LustreError: 197000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784942404 with bad export cookie 17976360607169225781 [ 7102.909694] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7102.916992] LustreError: 197000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7103.182240] Lustre: server umount lustre-MDT0001 complete [ 7109.048173] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 21:20:08 (1784942408) [ 7121.122945] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 7127.866817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7128.111674] LustreError: 202095:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7128.152611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7131.173377] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7133.151889] LustreError: 202096:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7136.746537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7140.330415] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7142.419325] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7147.195501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7151.815659] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7157.799677] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7157.922269] Lustre: lustre-OST0001: new disk, initializing [ 7157.929183] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7157.936652] Lustre: 204401:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7162.473169] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7168.029783] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7168.037627] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7168.134549] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7170.730627] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7175.117063] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7176.521646] Lustre: DEBUG MARKER: ADD OST3 [ 7179.943565] loop: module loaded [ 7179.998612] loop0: detected capacity change from 0 to 204800000 [ 7180.222640] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7180.231138] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7180.242038] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7184.436097] loop0: detected capacity change from 0 to 204800000 [ 7186.118662] Lustre: DEBUG MARKER: STOP OST3 [ 7188.456118] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7188.611128] Lustre: lustre-OST0003: new disk, initializing [ 7188.616245] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7188.621820] Lustre: 206331:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7188.730135] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7188.737668] Lustre: Skipped 3 previous similar messages [ 7189.786476] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7189.874682] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7189.879986] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7189.908770] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7190.237962] Lustre: Failing over lustre-OST0003 [ 7190.344711] Lustre: server umount lustre-OST0003 complete [ 7190.596462] LustreError: 203579:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.206.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7191.840093] LustreError: 206476:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7193.686564] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7195.115894] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7195.119784] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7195.129456] Lustre: Skipped 1 previous similar message [ 7200.229361] LustreError: 203581:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7200.236560] LustreError: 203581:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 7201.410830] Lustre: DEBUG MARKER: START OST3 again [ 7206.116443] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7206.296070] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7208.229948] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7208.246947] Lustre: lustre-OST0003: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 7208.247367] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7209.581089] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7226.048719] Lustre: Failing over lustre-OST0003 [ 7226.103287] Lustre: server umount lustre-OST0003 complete [ 7226.659848] LustreError: 207584:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.206.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7226.675436] LustreError: 207584:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 7226.854921] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7226.862140] Lustre: Skipped 2 previous similar messages [ 7242.213683] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7242.217187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7242.222142] Lustre: Skipped 3 previous similar messages [ 7242.225891] Lustre: Skipped 3 previous similar messages [ 7246.738433] Lustre: server umount lustre-MDT0000 complete [ 7248.726357] LustreError: 202077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784942550 with bad export cookie 17976360607169226663 [ 7248.728206] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7248.732547] LustreError: 202077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7263.199138] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7263.352423] Lustre: server umount lustre-MDT0001 complete [ 7265.527496] Lustre: server umount lustre-OST0000 complete [ 7267.629567] Lustre: server umount lustre-OST0001 complete [ 7271.528655] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 7276.240554] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 7298.328064] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7298.463082] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7298.486390] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7298.535905] Lustre: lustre-MDT0000: new disk, initializing [ 7298.596136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7298.599193] Lustre: Skipped 1 previous similar message [ 7298.609491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7301.445946] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7308.666553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7308.749421] Lustre: 211855:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7308.942228] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7311.741488] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7317.681115] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7320.985400] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7325.618600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7325.809184] Lustre: lustre-OST0000: new disk, initializing [ 7325.811690] Lustre: Skipped 1 previous similar message [ 7325.813942] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7325.817548] Lustre: Skipped 2 previous similar messages [ 7325.821176] Lustre: 213822:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7327.344526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7327.361851] Lustre: Skipped 1 previous similar message [ 7327.367825] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7327.429137] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7329.738105] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7336.434623] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7339.199892] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7339.322560] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7341.660604] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7341.800910] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7343.074974] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7343.075682] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7343.078507] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7343.078514] Lustre: Skipped 7 previous similar messages [ 7343.099522] Lustre: Skipped 3 previous similar messages [ 7356.895701] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7357.000339] Lustre: server umount lustre-OST0000 complete [ 7359.968249] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7359.973596] LustreError: Skipped 1 previous similar message [ 7365.266495] Lustre: server umount lustre-MDT0000 complete [ 7367.164427] LustreError: 212650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784942669 with bad export cookie 17976360607169228952 [ 7367.169065] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7367.170111] LustreError: 212650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7367.367750] Lustre: server umount lustre-MDT0001 complete [ 7369.327984] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 21:24:28 (1784942668) [ 7378.476207] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 7384.202317] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 7405.580326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7405.729232] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7405.748481] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7405.808404] Lustre: lustre-MDT0000: new disk, initializing [ 7405.864599] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7408.379493] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7415.159176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7415.231043] Lustre: 218427:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7415.401679] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7417.717938] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7422.916081] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7425.469467] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7428.838399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7428.995095] Lustre: 220392:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7429.065835] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7429.070372] Lustre: Skipped 4 previous similar messages [ 7430.510824] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7432.298944] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7437.451621] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7439.881611] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7440.011705] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7442.477661] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7442.622673] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7445.986495] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7445.992475] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7445.998957] Lustre: Skipped 2 previous similar messages [ 7446.002212] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7446.005701] Lustre: Skipped 8 previous similar messages [ 7449.718905] Lustre: server umount lustre-OST0000 complete [ 7457.962492] Lustre: server umount lustre-MDT0000 complete [ 7459.810854] LustreError: 218419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784942761 with bad export cookie 17976360607169229834 [ 7459.813677] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7459.816508] LustreError: 218419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7459.979718] Lustre: server umount lustre-MDT0001 complete [ 7464.069127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7464.296213] LustreError: 222606:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7464.305922] LustreError: 222606:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 18 previous similar messages [ 7466.336133] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7470.588228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7472.475332] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7475.579595] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7477.903536] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7481.068462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7484.054537] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7487.334715] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7499.744187] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7499.748936] LustreError: Skipped 1 previous similar message [ 7501.802906] Lustre: server umount lustre-OST0000 complete [ 7505.658752] Lustre: server umount lustre-MDT0000 complete [ 7507.400356] LustreError: 222587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784942809 with bad export cookie 17976360607169230716 [ 7507.402455] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7507.406600] LustreError: 222587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7507.568556] Lustre: server umount lustre-MDT0001 complete [ 7622.222841] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 7623.540793] Key type lgssc unregistered [ 7623.716350] LNet: 226288:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7623.720720] LNetError: 226288:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7623.729494] LNet: Removed LNI 192.168.206.151@tcp [ 7624.118145] Key type .llcrypt unregistered [ 7624.119900] Key type ._llcrypt unregistered [ 7635.461920] Key type ._llcrypt registered [ 7635.463277] Key type .llcrypt registered [ 7635.526982] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 7643.726709] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 7644.102353] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7644.150396] alg: No test for adler32 (adler32-zlib) [ 7645.059483] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [ 7645.171832] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 7646.783260] Key type lgssc registered [ 7647.306814] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7663.850968] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7663.872935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7665.001804] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7665.017899] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7665.064922] Lustre: lustre-MDT0000: new disk, initializing [ 7665.107558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7665.116619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7667.115655] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7672.570639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7672.639567] Lustre: 229614:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7672.686780] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7672.689423] Lustre: Skipped 1 previous similar message [ 7672.734647] Lustre: lustre-MDT0001: new disk, initializing [ 7672.785725] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7672.807610] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7672.814048] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7674.603599] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7679.459800] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7682.173217] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7685.369287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7685.526043] Lustre: lustre-OST0000: new disk, initializing [ 7685.531096] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7685.536918] Lustre: 231583:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7685.592974] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7688.193337] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7693.093141] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7693.327672] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7693.332966] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7693.384095] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7695.145850] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7695.258819] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7697.141785] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7697.248943] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7698.401087] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7698.402650] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7698.405093] LustreError: Skipped 1 previous similar message [ 7698.410757] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7698.415214] Lustre: Skipped 1 previous similar message [ 7700.253989] Lustre: server umount lustre-OST0000 complete [ 7703.521314] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7703.530759] Lustre: Skipped 1 previous similar message [ 7703.533703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7703.536056] Lustre: Skipped 1 previous similar message [ 7708.640785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7708.644086] Lustre: Skipped 1 previous similar message [ 7713.760585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7713.763850] Lustre: Skipped 1 previous similar message [ 7716.319128] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7716.387413] Lustre: server umount lustre-MDT0000 complete [ 7718.001415] LustreError: 229609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784943019 with bad export cookie 16298394852113350203 [ 7718.003548] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7718.010461] LustreError: 229609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7718.231576] Lustre: server umount lustre-MDT0001 complete [ 7726.368305] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 7730.847754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7731.049876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7732.695577] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7736.207359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7736.287879] LustreError: 234510:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7736.293671] LustreError: 234510:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7736.382475] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7738.077477] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7739.499152] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7742.549194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7745.282745] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7749.309518] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7749.393531] Lustre: lustre-OST0001: new disk, initializing [ 7749.396652] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7749.401445] Lustre: 236817:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7749.444857] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7749.448849] Lustre: Skipped 1 previous similar message [ 7751.202352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7751.206938] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7751.256411] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7752.024779] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7758.221936] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7761.493268] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7767.168355] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 21:31:06 (1784943066) [ 7770.209319] Lustre: server umount lustre-MDT0000 complete [ 7771.107742] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7771.117027] Lustre: Skipped 3 previous similar messages [ 7771.119634] LustreError: 234505:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7771.127328] LustreError: 234505:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7771.818396] LustreError: 234491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784943073 with bad export cookie 16298394852113351071 [ 7771.822372] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7771.823085] LustreError: 234491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7771.973788] Lustre: server umount lustre-MDT0001 complete [ 7783.446855] Lustre: server umount lustre-OST0000 complete [ 7794.298708] Lustre: server umount lustre-OST0001 complete [ 7796.981680] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 7801.308826] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 7815.731786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7815.832058] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7815.847470] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7815.890527] Lustre: lustre-MDT0000: new disk, initializing [ 7815.931236] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7815.944675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7817.584871] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7822.349402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7822.398704] Lustre: 240751:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7822.517760] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7824.083446] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7828.329958] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7830.608484] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7833.241711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7833.346790] Lustre: lustre-OST0000: new disk, initializing [ 7833.348634] Lustre: Skipped 1 previous similar message [ 7833.351165] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7833.354128] Lustre: Skipped 2 previous similar messages [ 7833.356909] Lustre: 242719:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7834.429947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7834.433514] Lustre: Skipped 1 previous similar message [ 7834.435708] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7834.469518] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7835.606881] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7839.873414] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7841.529124] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7841.613587] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7843.239727] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7843.320083] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7844.833709] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7844.835120] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7844.835589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7844.839117] Lustre: Skipped 2 previous similar messages [ 7844.842268] LustreError: Skipped 1 previous similar message [ 7850.144960] Lustre: server umount lustre-OST0000 complete [ 7853.539695] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7853.544926] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7853.550619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7853.552871] Lustre: Skipped 3 previous similar messages [ 7866.335113] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7866.404437] Lustre: server umount lustre-MDT0000 complete [ 7868.103353] LustreError: 240744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784943170 with bad export cookie 16298394852113353164 [ 7868.106925] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7868.110977] LustreError: 240744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7868.269290] Lustre: server umount lustre-MDT0001 complete [ 7871.921957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7872.143074] LustreError: 244934:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7872.186590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7872.190021] Lustre: Skipped 2 previous similar messages [ 7873.891834] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7877.356838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7879.079635] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7882.002391] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7884.046942] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7886.507787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7888.776670] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7891.599912] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8036.327450] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8036.327672] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8036.331471] LustreError: Skipped 1 previous similar message [ 8036.338183] Lustre: Skipped 2 previous similar messages [ 8036.342351] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8036.347071] Lustre: Skipped 7 previous similar messages [ 8042.401474] Lustre: server umount lustre-OST0000 complete [ 8046.561033] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8058.336036] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8058.412204] Lustre: server umount lustre-MDT0000 complete [ 8060.199737] LustreError: 244915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784943362 with bad export cookie 16298394852113354046 [ 8060.202287] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8060.206277] LustreError: 244915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8060.377810] Lustre: server umount lustre-MDT0001 complete [ 8158.564154] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 8159.850983] Key type lgssc unregistered [ 8160.007382] LNet: 248538:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8160.010543] LNetError: 248538:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8161.062180] LNet: Removed LNI 192.168.206.151@tcp [ 8161.480233] Key type .llcrypt unregistered [ 8161.482451] Key type ._llcrypt unregistered [ 8172.010928] Key type ._llcrypt registered [ 8172.012697] Key type .llcrypt registered [ 8172.065116] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 8179.638631] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 8180.054480] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8180.094077] alg: No test for adler32 (adler32-zlib) [ 8180.990486] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [ 8181.123083] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 8182.727150] Key type lgssc registered [ 8183.234939] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8202.882210] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 8207.303719] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8207.312696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8208.434711] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8208.457170] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8208.509595] Lustre: lustre-MDT0000: new disk, initializing [ 8208.552423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8208.564093] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8210.329381] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8215.752963] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8215.805591] Lustre: 252580:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8215.849913] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8215.852265] Lustre: Skipped 1 previous similar message [ 8215.894928] Lustre: lustre-MDT0001: new disk, initializing [ 8215.939683] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8215.968067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8215.978216] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8217.994730] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8221.072562] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8224.311662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8224.464786] Lustre: lustre-OST0000: new disk, initializing [ 8224.467447] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8224.471280] Lustre: 254210:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8224.524752] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8225.354535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8225.361568] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8225.423810] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8227.420677] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8232.994197] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8233.084729] Lustre: lustre-OST0001: new disk, initializing [ 8233.087578] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8233.090929] Lustre: 255084:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8233.150445] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8236.060220] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8242.214351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8242.223980] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8242.288412] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8242.558323] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8247.953420] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8255.165529] Lustre: server umount lustre-MDT0000 complete [ 8256.994456] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8256.997785] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8257.008936] LustreError: 252592:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8257.015945] LustreError: 252592:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8257.231313] LustreError: 255083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784943559 with bad export cookie 12326026426393788364 [ 8257.233403] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8257.239916] LustreError: 255083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8257.505255] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8257.506851] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8257.510443] Lustre: Skipped 3 previous similar messages [ 8259.441164] Lustre: server umount lustre-MDT0001 complete [ 8263.383840] Lustre: server umount lustre-OST0000 complete [ 8265.058414] Lustre: server umount lustre-OST0001 complete [ 8270.743207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8274.849884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8278.408487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 8282.497624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 8290.342551] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 8294.051031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8294.060873] Lustre: lustre-MDT0000: reset Object Index mappings [ 8294.228517] LustreError: 260080:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8294.238172] LustreError: 260080:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8294.252360] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8295.753438] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8299.085384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8299.094987] Lustre: lustre-MDT0001: reset Object Index mappings [ 8300.841659] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8302.050985] Lustre: 261219:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8304.974154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8305.146339] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8305.149340] Lustre: Skipped 1 previous similar message [ 8307.557210] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8309.221743] LustreError: 261575:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8311.147585] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8313.866405] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8319.470935] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 8319.471102] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8322.416516] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8323.842995] Lustre: 263090:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8329.698038] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8329.704769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8329.709365] Lustre: Skipped 4 previous similar messages [ 8331.500625] Lustre: server umount lustre-MDT0000 complete [ 8333.198360] LustreError: 263092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784943635 with bad export cookie 12326026426393795126 [ 8333.202879] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8333.203389] LustreError: 263092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8333.344580] Lustre: server umount lustre-MDT0001 complete [ 8344.149269] Lustre: server umount lustre-OST0000 complete [ 8356.004797] Lustre: server umount lustre-OST0001 complete [ 8363.522093] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 21:41:03 (1784943663) [ 8369.727100] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 8373.719098] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 8386.868398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8386.956820] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8386.971298] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8387.011435] Lustre: lustre-MDT0000: new disk, initializing [ 8387.036677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8387.038948] Lustre: Skipped 1 previous similar message [ 8387.044542] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8388.595141] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8393.026739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8393.071134] Lustre: 267335:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8393.176021] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8394.646703] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8398.701154] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8400.838366] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8403.539240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8403.657535] Lustre: lustre-OST0000: new disk, initializing [ 8403.659077] Lustre: Skipped 1 previous similar message [ 8403.660937] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8403.662847] Lustre: Skipped 2 previous similar messages [ 8403.666381] Lustre: 269304:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8405.183628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8405.190372] Lustre: Skipped 1 previous similar message [ 8405.192750] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8405.238823] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8406.017396] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8410.186617] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8411.891316] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8411.980752] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8413.651921] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8413.741767] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8415.713396] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8415.713917] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8415.716022] LustreError: Skipped 1 previous similar message [ 8415.720567] Lustre: Skipped 3 previous similar messages [ 8415.722909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8415.725944] Lustre: Skipped 1 previous similar message [ 8420.648070] Lustre: server umount lustre-OST0000 complete [ 8423.907978] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8423.911259] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8423.916994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8428.250142] Lustre: server umount lustre-MDT0000 complete [ 8429.713189] LustreError: 267329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784943731 with bad export cookie 12326026426393797212 [ 8429.716695] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8429.719704] LustreError: 267329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8429.845986] Lustre: server umount lustre-MDT0001 complete [ 8436.963272] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 8440.877582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8441.016455] LustreError: 272231:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8441.032476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8441.034476] Lustre: Skipped 2 previous similar messages [ 8442.521427] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8445.866216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8447.589421] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8448.704819] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8451.348114] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8453.679826] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8456.873231] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8456.943502] Lustre: lustre-OST0001: new disk, initializing [ 8456.945585] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8456.949102] Lustre: 274533:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8458.124801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8458.129085] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8458.182726] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8459.297626] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8464.069451] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8465.466498] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8467.293367] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 8467.298599] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 8470.498831] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8470.500597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8470.503343] Lustre: Skipped 3 previous similar messages [ 8470.505371] Lustre: Skipped 3 previous similar messages [ 8475.097804] Lustre: server umount lustre-MDT0000 complete [ 8475.616995] LustreError: 272232:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8475.624053] LustreError: 272232:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8476.310176] LustreError: 272965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784943778 with bad export cookie 12326026426393798094 [ 8476.313575] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8476.314490] LustreError: 272965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8476.425706] Lustre: server umount lustre-MDT0001 complete [ 8487.938790] Lustre: server umount lustre-OST0000 complete [ 8499.494609] Lustre: server umount lustre-OST0001 complete [ 8505.103313] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 8508.933389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8509.100049] LustreError: 277586:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8509.118796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8509.121368] Lustre: Skipped 3 previous similar messages [ 8510.602471] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8513.590197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8515.191500] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8516.235787] Lustre: 278727:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8518.615586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8519.783072] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 8520.810493] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8523.960040] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8526.117700] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8529.387473] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 8529.473524] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8530.783611] Lustre: 280600:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8532.705306] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 8532.707118] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 8535.816157] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 21:43:55 (1784943835) [ 8549.861663] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8549.862855] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8549.864350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8549.864362] Lustre: Skipped 2 previous similar messages [ 8549.866678] Lustre: Skipped 3 previous similar messages [ 8553.429812] Lustre: server umount lustre-MDT0000 complete [ 8554.713346] LustreError: 277566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784943856 with bad export cookie 12326026426393804401 [ 8554.716156] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8554.717924] LustreError: 277566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8554.977681] LustreError: 277581:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8554.983101] LustreError: 277581:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 8569.311102] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8569.312867] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8569.318461] Lustre: Skipped 10 previous similar messages [ 8569.399955] Lustre: server umount lustre-MDT0001 complete [ 8576.765640] Lustre: server umount lustre-OST0000 complete [ 8592.351106] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8592.415832] Lustre: server umount lustre-OST0001 complete [ 8594.695295] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 8598.298561] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 8614.755527] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 8618.450976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8618.527736] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8618.537124] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8618.569802] Lustre: lustre-MDT0000: new disk, initializing [ 8618.592946] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8620.004565] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8624.238327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8624.284794] Lustre: 284832:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8624.381748] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8625.833853] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8630.387069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8630.490886] Lustre: 286464:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8631.276446] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8632.599215] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8636.712900] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8637.855590] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8638.686516] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8643.340996] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8648.827649] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 21:45:48 (1784943948) [ 8650.207845] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8650.210580] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8650.215664] Lustre: Skipped 3 previous similar messages [ 8650.217621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8655.645372] Lustre: server umount lustre-MDT0000 complete [ 8656.800156] LustreError: 284823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784943958 with bad export cookie 12326026426393812283 [ 8656.802669] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8656.803842] LustreError: 284823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8656.911123] Lustre: server umount lustre-MDT0001 complete [ 8668.611236] Lustre: server umount lustre-OST0000 complete [ 8680.090307] Lustre: server umount lustre-OST0001 complete [ 8682.633668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8682.765958] LustreError: 289573:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8682.772137] LustreError: 289573:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 8682.795317] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8682.798182] Lustre: Skipped 7 previous similar messages [ 8684.077661] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8686.845179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8688.335600] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8690.779602] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8692.688633] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8694.774361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8696.697877] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8698.900529] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8714.989617] loop1: detected capacity change from 0 to 204800000 [ 8715.067433] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8715.071124] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8715.074946] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8716.639344] loop1: detected capacity change from 0 to 204800000 [ 8718.370088] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8718.401286] LustreError: 289562:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs29379: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 8718.406686] LustreError: 289562:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs29379 [ 8718.408604] LustreError: 289562:0:(mgs_handler.c:517:mgs_target_reg()) fs29379-MDT0001: Error checking target index: rc = -13 [ 8718.412705] LustreError: fs29379-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 8718.416826] LustreError: 293310:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 8718.419728] LustreError: 293310:0:(tgt_mount.c:2129:server_put_super()) no obd fs29379-MDT0001 [ 8718.422153] LustreError: 293310:0:(tgt_mount.c:116:server_deregister_mount()) fs29379-MDT0001 not registered [ 8718.433605] Lustre: server umount fs29379-MDT0001 complete [ 8718.435625] LustreError: 293310:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 8721.002985] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8721.029034] Lustre: Setting parameter fs29379-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs29379-MDT0001 [ 8721.031574] Lustre: Skipped 2 previous similar messages [ 8721.044717] Lustre: srv-fs29379-MDT0001: No data found on store. Initialize space. [ 8721.047123] Lustre: Skipped 4 previous similar messages [ 8721.081869] Lustre: fs29379-MDT0001: new disk, initializing [ 8721.083170] Lustre: Skipped 3 previous similar messages [ 8721.118530] Lustre: srv-fs29379-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 8721.120534] Lustre: 293775:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs29379-MDT0001-mdtlov: can't create new llog: rc = -115 [ 8721.122698] LustreError: 293775:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs29379-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 8722.367020] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8725.165210] Lustre: server umount fs29379-MDT0001 complete [ 8729.121529] loop1: detected capacity change from 0 to 204800000 [ 8730.806273] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8730.908307] Lustre: 295342:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs29379-MDT0001-mdtlov: can't create new llog: rc = -115 [ 8730.910615] LustreError: 295342:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs29379-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 8732.113196] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8733.681312] Lustre: server umount fs29379-MDT0001 complete [ 8738.275610] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8738.276373] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8738.281030] Lustre: Skipped 3 previous similar messages [ 8738.283524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8738.286277] Lustre: Skipped 6 previous similar messages [ 8742.994076] Lustre: server umount lustre-MDT0000 complete [ 8745.549170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8745.584918] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8746.876312] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8749.127685] loop1: detected capacity change from 0 to 204800000 [ 8750.350219] LustreError: 294823:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8750.358310] LustreError: 294823:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 30 previous similar messages [ 8750.841788] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8750.948039] Lustre: 297783:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs29379-MDT0001-mdtlov: can't create new llog: rc = -115 [ 8750.950795] LustreError: 297783:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs29379-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 8751.077356] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8752.179574] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8753.681283] Lustre: server umount fs29379-MDT0001 complete [ 8761.315899] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8766.036588] Lustre: server umount lustre-OST0000 complete [ 8773.333165] Lustre: server umount lustre-MDT0000 complete [ 8774.453312] LustreError: 289554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784944076 with bad export cookie 12326026426393814957 [ 8774.456075] LustreError: 289554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8774.580584] Lustre: server umount lustre-MDT0001 complete [ 8777.802118] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 8778.736293] Key type lgssc unregistered [ 8778.868445] LNet: 300121:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8778.871907] LNetError: 300121:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8778.880340] LNet: Removed LNI 192.168.206.151@tcp [ 8779.178114] Key type .llcrypt unregistered [ 8779.179301] Key type ._llcrypt unregistered [ 8786.343333] Key type ._llcrypt registered [ 8786.344650] Key type .llcrypt registered [ 8786.382782] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 8786.693714] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8786.751045] alg: No test for adler32 (adler32-zlib) [ 8787.603402] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [ 8787.692377] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 8789.271130] Key type lgssc registered [ 8789.600812] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8791.948757] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8791.956404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8793.083348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8794.304548] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8797.010872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8797.181567] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8798.594986] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8800.943058] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8802.679108] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8804.901045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8805.025077] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8806.825581] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8807.076144] LustreError: 303592:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8807.082436] LustreError: 303592:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8809.099955] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8812.515291] LustreError: 303194:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8817.634851] LustreError: 303196:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8823.779469] LustreError: 303195:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8823.786753] LustreError: 303195:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 8824.981823] loop1: detected capacity change from 0 to 204800000 [ 8825.037960] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8825.041516] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8825.044913] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8826.588449] loop1: detected capacity change from 0 to 204800000 [ 8828.264094] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8828.314603] Lustre: fs11295-OST0000: new disk, initializing [ 8828.316304] Lustre: srv-fs11295-OST0000: No data found on store. Initialize space. [ 8828.319118] Lustre: 305123:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8828.338622] Lustre: fs11295-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8828.898687] LustreError: 303195:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8828.902173] LustreError: 303195:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8830.050563] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8833.237125] Lustre: server umount fs11295-OST0000 complete [ 8836.939893] loop1: detected capacity change from 0 to 204800000 [ 8838.637696] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8838.687957] Lustre: fs11295-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8839.138962] LustreError: 303195:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8839.142679] LustreError: 303195:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 8840.367952] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8842.940938] loop2: detected capacity change from 0 to 204800000 [ 8843.024364] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8843.028571] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8843.032529] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8844.584870] loop2: detected capacity change from 0 to 204800000 [ 8846.307649] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8846.336689] LustreError: fs11295-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 8846.339977] LustreError: fs11295-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 8846.342838] LustreError: 308159:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 8846.344979] LustreError: 308159:0:(tgt_mount.c:2129:server_put_super()) no obd fs11295-OST0001 [ 8846.346901] LustreError: 308159:0:(tgt_mount.c:116:server_deregister_mount()) fs11295-OST0001 not registered [ 8846.356886] Lustre: server umount fs11295-OST0001 complete [ 8846.357948] LustreError: 308159:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 8849.551550] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8849.590330] Lustre: fs11295-OST0001: new disk, initializing [ 8849.591759] Lustre: srv-fs11295-OST0001: No data found on store. Initialize space. [ 8849.594217] Lustre: 308709:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8849.613700] Lustre: fs11295-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8851.389740] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8854.689514] Lustre: server umount fs11295-OST0000 complete [ 8855.309138] LustreError: 303196:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8855.314664] LustreError: 303196:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 8857.482919] Lustre: server umount fs11295-OST0001 complete [ 8861.161330] loop1: detected capacity change from 0 to 204800000 [ 8862.869580] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8864.677245] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8866.719615] loop2: detected capacity change from 0 to 204800000 [ 8868.414907] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8868.474105] Lustre: fs11295-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8868.475941] Lustre: Skipped 1 previous similar message [ 8870.224162] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8872.905968] Lustre: server umount fs11295-OST0000 complete [ 8875.656811] Lustre: server umount fs11295-OST0001 complete [ 8879.072297] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8879.075228] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8879.080764] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8880.097148] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8880.097201] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8880.098588] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8885.003561] Lustre: server umount lustre-OST0000 complete [ 8889.311762] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8889.314236] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8889.318185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8889.319547] Lustre: Skipped 1 previous similar message [ 8892.244121] Lustre: server umount lustre-MDT0000 complete [ 8893.454145] LustreError: 303219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784944195 with bad export cookie 869943536450912016 [ 8893.457418] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8893.458284] LustreError: 303219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8893.571603] Lustre: server umount lustre-MDT0001 complete [ 8896.827701] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 8897.723039] Key type lgssc unregistered [ 8897.830341] LNet: 314618:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8897.832509] LNetError: 314618:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8897.839272] LNet: Removed LNI 192.168.206.151@tcp [ 8898.124085] Key type .llcrypt unregistered [ 8898.124947] Key type ._llcrypt unregistered [ 8907.465084] Key type ._llcrypt registered [ 8907.466031] Key type .llcrypt registered [ 8907.504180] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 8913.323289] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 8913.619866] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8913.650023] alg: No test for adler32 (adler32-zlib) [ 8914.490869] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [ 8914.575184] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 8916.151121] Key type lgssc registered [ 8916.475113] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8927.088087] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8927.093913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8928.164186] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8928.173489] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8928.209573] Lustre: lustre-MDT0000: new disk, initializing [ 8928.230567] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8928.237267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8929.470631] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8933.431940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8933.475107] Lustre: 318445:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8933.499299] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8933.501148] Lustre: Skipped 1 previous similar message [ 8933.539356] Lustre: lustre-MDT0001: new disk, initializing [ 8933.576590] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8933.590701] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8933.594157] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8934.838853] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8938.410497] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8940.062464] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8942.176084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8942.271291] Lustre: lustre-OST0000: new disk, initializing [ 8942.273215] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8942.276216] Lustre: 320415:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8942.302526] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8944.190423] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8947.734859] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8950.262222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8950.264702] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8950.290860] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8955.377494] Lustre: Mounted lustre-client [ 8955.685571] Lustre: Unmounted lustre-client [ 8960.480613] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8960.482960] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8960.483582] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8960.483755] LustreError: Skipped 1 previous similar message [ 8960.487364] Lustre: Skipped 1 previous similar message [ 8964.491700] Lustre: server umount lustre-OST0000 complete [ 8969.695754] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8969.698155] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8969.702791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8969.705098] Lustre: Skipped 1 previous similar message [ 8970.720523] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8970.720556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8970.726908] Lustre: Skipped 1 previous similar message [ 8971.793790] Lustre: server umount lustre-MDT0000 complete [ 8972.941030] LustreError: 318439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784944274 with bad export cookie 7153716510708165014 [ 8972.944116] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8972.946524] LustreError: 318439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8973.054083] Lustre: server umount lustre-MDT0001 complete [ 8976.506772] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 8977.490739] Key type lgssc unregistered [ 9007.327398] LNet: 322574:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9007.329829] LNetError: 322574:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9077.989302] LNet: Removed LNI 192.168.206.151@tcp [ 9078.318299] Key type .llcrypt unregistered [ 9078.319954] Key type ._llcrypt unregistered [ 9084.340228] Key type ._llcrypt registered [ 9084.341157] Key type .llcrypt registered [ 9084.385933] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 9090.457409] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 9090.894489] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9090.905613] alg: No test for adler32 (adler32-zlib) [ 9091.760675] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [ 9091.841780] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 9093.415160] Key type lgssc registered [ 9093.784725] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9103.289883] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 21:53:23 (1784944403) [ 9105.352665] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9105.357697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9106.432740] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9106.441914] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9106.477432] Lustre: lustre-MDT0000: new disk, initializing [ 9106.496429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9106.501558] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9107.726208] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9111.537137] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9111.575781] Lustre: 325768:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9111.600352] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9111.601916] Lustre: Skipped 1 previous similar message [ 9111.637137] Lustre: lustre-MDT0001: new disk, initializing [ 9111.668095] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9111.680173] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9111.683565] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9112.951418] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9116.482240] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9118.157405] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9120.327727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9120.427132] Lustre: lustre-OST0000: new disk, initializing [ 9120.428670] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9120.430750] Lustre: 327737:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9120.459794] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9121.847533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9121.851968] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9121.881770] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9122.332336] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9125.858351] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9132.000941] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9132.002601] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9132.008273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9132.009850] Lustre: Skipped 1 previous similar message [ 9133.136461] Lustre: server umount lustre-OST0000 complete [ 9136.342765] Lustre: server umount lustre-MDT0000 complete [ 9137.120310] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9137.124718] Lustre: Skipped 2 previous similar messages [ 9137.485958] LustreError: 327736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784944439 with bad export cookie 10997785830793568959 [ 9137.488401] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9137.488823] LustreError: 327736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9137.601199] Lustre: server umount lustre-MDT0001 complete [ 9140.875969] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 9141.777876] Key type lgssc unregistered [ 9141.888367] LNet: 329641:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9141.890864] LNetError: 329641:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9141.897322] LNet: Removed LNI 192.168.206.151@tcp [ 9142.214096] Key type .llcrypt unregistered [ 9142.215799] Key type ._llcrypt unregistered [ 9149.668963] Key type ._llcrypt registered [ 9149.669973] Key type .llcrypt registered [ 9149.705827] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 9149.976896] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9149.982841] alg: No test for adler32 (adler32-zlib) [ 9150.825265] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [ 9150.901682] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 9152.479149] Key type lgssc registered [ 9152.822118] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9155.167379] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9155.172641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9156.298967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9157.490569] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9160.124803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9160.248993] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9161.485826] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9163.815072] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9165.499162] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9167.549281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9167.655979] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9169.386823] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9171.511521] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9190.123172] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9190.612185] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9192.210690] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.51@tcp (stopping) [ 9192.416550] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9192.420539] Lustre: Skipped 2 previous similar messages [ 9193.933512] Lustre: server umount lustre-MDT0000 complete [ 9195.021145] LustreError: 330916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784944496 with bad export cookie 7987198062751812884 [ 9195.023763] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9195.024264] LustreError: 330916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9197.325034] LustreError: 330887:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9197.330327] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.51@tcp (stopping) [ 9197.330660] LustreError: 330887:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9197.332498] Lustre: Skipped 3 previous similar messages [ 9197.536183] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9200.543736] LustreError: 332328:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9200.543880] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9200.548227] LustreError: 332328:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9200.549643] Lustre: Skipped 1 previous similar message [ 9201.190097] Lustre: server umount lustre-MDT0001 complete [ 9207.565059] Lustre: lustre-OST0000: Not available for connect from 192.168.206.51@tcp (stopping) [ 9207.567460] Lustre: Skipped 1 previous similar message [ 9208.541144] Lustre: server umount lustre-OST0000 complete [ 9211.377421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9212.805405] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9213.761284] Lustre: 335088:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.151@tcp [ 9214.389754] Lustre: server umount MGS complete [ 9216.877892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9217.008962] LustreError: 335658:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9217.029393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9218.261832] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9220.761282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9222.055786] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9224.658952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9224.759419] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9224.762436] Lustre: Skipped 1 previous similar message [ 9226.530387] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9228.780828] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9234.322544] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 5 sec [ 9235.993879] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9236.478694] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9241.056489] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9241.058586] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9241.060020] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9241.062926] Lustre: Skipped 1 previous similar message [ 9241.065660] Lustre: Skipped 1 previous similar message [ 9244.296134] Lustre: server umount lustre-OST0000 complete [ 9246.176351] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9246.179275] Lustre: Skipped 1 previous similar message [ 9251.295893] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9251.298114] Lustre: Skipped 3 previous similar messages [ 9251.533702] Lustre: server umount lustre-MDT0000 complete [ 9252.620960] LustreError: 335638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784944554 with bad export cookie 7987198062751813710 [ 9252.624644] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9252.625077] LustreError: 335638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9252.722805] Lustre: server umount lustre-MDT0001 complete [ 9255.926852] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 9256.878262] Key type lgssc unregistered [ 9257.011341] LNet: 339092:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9257.013553] LNetError: 339092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9257.024397] LNet: Removed LNI 192.168.206.151@tcp [ 9257.312477] Key type .llcrypt unregistered [ 9257.313434] Key type ._llcrypt unregistered [ 9262.907037] Key type ._llcrypt registered [ 9262.908543] Key type .llcrypt registered [ 9262.941632] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 9269.033051] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 9269.290675] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9269.318891] alg: No test for adler32 (adler32-zlib) [ 9270.161375] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [ 9270.240646] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 9271.815096] Key type lgssc registered [ 9272.137466] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9282.820278] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9282.825850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9283.907387] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9283.919114] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9283.961663] Lustre: lustre-MDT0000: new disk, initializing [ 9283.985918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9283.994677] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9285.433301] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9289.466466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9289.505350] Lustre: 342225:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9289.525484] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9289.526966] Lustre: Skipped 1 previous similar message [ 9289.557101] Lustre: lustre-MDT0001: new disk, initializing [ 9289.586095] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9289.596511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9289.599683] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9290.747252] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9294.097057] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9295.694989] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9297.793160] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9297.902733] Lustre: lustre-OST0000: new disk, initializing [ 9297.904311] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9297.906798] Lustre: 344193:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9297.934765] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9299.671841] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9301.241847] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9301.246889] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9301.280425] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9303.558707] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9305.014248] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9305.090363] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9306.435284] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9306.509085] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9311.713156] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9311.714712] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9311.715489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9311.716140] LustreError: Skipped 1 previous similar message [ 9311.719646] Lustre: Skipped 1 previous similar message [ 9313.301150] Lustre: server umount lustre-OST0000 complete [ 9315.295737] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9315.297387] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9315.300341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9315.301765] Lustre: Skipped 1 previous similar message [ 9316.832284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9316.832833] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9320.593769] Lustre: server umount lustre-MDT0000 complete [ 9321.720086] LustreError: 344192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784944623 with bad export cookie 9105473816996889055 [ 9321.722792] LustreError: 344192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9321.723197] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9321.822694] Lustre: server umount lustre-MDT0001 complete [ 9324.422145] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 21:57:04 (1784944624) [ 9329.417605] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 9332.492849] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 9343.741425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9343.827348] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9343.838619] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9343.877340] Lustre: lustre-MDT0000: new disk, initializing [ 9343.898112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9343.906064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9345.235877] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9349.294773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9349.338245] Lustre: 348942:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9349.362206] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9349.364933] Lustre: Skipped 1 previous similar message [ 9349.406270] Lustre: lustre-MDT0001: new disk, initializing [ 9349.440092] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9349.451995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9349.456199] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9350.859270] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9354.649465] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9356.500601] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9358.791614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9358.889153] Lustre: lustre-OST0000: new disk, initializing [ 9358.891206] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9358.894591] Lustre: 350910:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9358.921807] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9360.816777] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9364.646023] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9368.565866] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9368.569040] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9368.603968] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9378.783919] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9378.784308] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9378.787287] LustreError: Skipped 1 previous similar message [ 9378.791213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9378.794387] Lustre: Skipped 2 previous similar messages [ 9380.551347] Lustre: server umount lustre-OST0000 complete [ 9383.904663] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9383.905389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9383.908464] Lustre: Skipped 2 previous similar messages [ 9383.911575] Lustre: Skipped 1 previous similar message [ 9387.793072] Lustre: server umount lustre-MDT0000 complete [ 9388.967295] LustreError: 348933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784944690 with bad export cookie 9105473816996889937 [ 9388.971140] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9388.973589] LustreError: 348933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9389.087788] Lustre: server umount lustre-MDT0001 complete [ 9392.682391] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 9393.677588] Key type lgssc unregistered [ 9393.806407] LNet: 353056:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9393.810265] LNetError: 353056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9393.821307] LNet: Removed LNI 192.168.206.151@tcp [ 9394.126966] Key type .llcrypt unregistered [ 9394.128379] Key type ._llcrypt unregistered [ 9425.232372] Key type ._llcrypt registered [ 9425.233447] Key type .llcrypt registered [ 9425.272416] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 9425.385312] Key type .llcrypt unregistered [ 9425.386404] Key type ._llcrypt unregistered [ 9452.627144] Key type ._llcrypt registered [ 9452.628183] Key type .llcrypt registered [ 9452.666508] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 9452.765357] Key type .llcrypt unregistered [ 9452.766449] Key type ._llcrypt unregistered [ 9463.948969] Key type ._llcrypt registered [ 9463.950082] Key type .llcrypt registered [ 9463.986245] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 9464.080174] Key type .llcrypt unregistered [ 9464.081450] Key type ._llcrypt unregistered [ 9475.002278] Key type ._llcrypt registered [ 9475.003392] Key type .llcrypt registered [ 9475.045308] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 9475.160271] Key type .llcrypt unregistered [ 9475.161324] Key type ._llcrypt unregistered [ 9501.490076] Key type ._llcrypt registered [ 9501.491471] Key type .llcrypt registered [ 9501.524884] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 9501.616413] Key type .llcrypt unregistered [ 9501.617272] Key type ._llcrypt unregistered [ 9511.684407] Key type ._llcrypt registered [ 9511.685323] Key type .llcrypt registered [ 9511.719322] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 9511.810387] Key type .llcrypt unregistered [ 9511.811464] Key type ._llcrypt unregistered [ 9522.849955] Key type ._llcrypt registered [ 9522.851558] Key type .llcrypt registered [ 9522.893197] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 9522.989318] Key type .llcrypt unregistered [ 9522.990340] Key type ._llcrypt unregistered [ 9533.779729] Key type ._llcrypt registered [ 9533.781560] Key type .llcrypt registered [ 9533.820952] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 9533.921169] Key type .llcrypt unregistered [ 9533.922144] Key type ._llcrypt unregistered [ 9544.583100] Key type ._llcrypt registered [ 9544.584598] Key type .llcrypt registered [ 9544.619849] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 9544.716368] Key type .llcrypt unregistered [ 9544.717541] Key type ._llcrypt unregistered [ 9555.629404] Key type ._llcrypt registered [ 9555.630415] Key type .llcrypt registered [ 9555.666240] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 9555.761193] Key type .llcrypt unregistered [ 9555.762906] Key type ._llcrypt unregistered [ 9576.701074] Key type ._llcrypt registered [ 9576.702125] Key type .llcrypt registered [ 9576.740472] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 9576.846222] Key type .llcrypt unregistered [ 9576.847499] Key type ._llcrypt unregistered [ 9588.012934] Key type ._llcrypt registered [ 9588.014046] Key type .llcrypt registered [ 9588.051835] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 9588.155377] Key type .llcrypt unregistered [ 9588.156918] Key type ._llcrypt unregistered [ 9599.081803] Key type ._llcrypt registered [ 9599.082789] Key type .llcrypt registered [ 9599.118407] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 9599.227192] Key type .llcrypt unregistered [ 9599.228241] Key type ._llcrypt unregistered [ 9617.263653] Key type ._llcrypt registered [ 9617.265289] Key type .llcrypt registered [ 9617.304174] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 9623.540647] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 9623.912452] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9623.940786] alg: No test for adler32 (adler32-zlib) [ 9624.801290] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [ 9624.885691] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 9626.463123] Key type lgssc registered [ 9626.839780] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9638.423923] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9638.430164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9639.516322] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [ 9639.525401] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [ 9639.561718] Lustre: tst250fs-MDT0000: new disk, initializing [ 9639.582715] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9639.608581] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9640.978877] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9645.155626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9645.202473] Lustre: 372968:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9645.227995] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [ 9645.231013] Lustre: Skipped 1 previous similar message [ 9645.271706] Lustre: tst250fs-MDT0001: new disk, initializing [ 9645.307190] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9645.321687] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9645.324861] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9646.654343] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9648.896355] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9651.016839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9651.118695] Lustre: tst250fs-OST0000: new disk, initializing [ 9651.120338] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [ 9651.122569] Lustre: 374600:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9651.151231] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9653.001291] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9657.059708] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9657.128752] Lustre: tst250fs-OST0001: new disk, initializing [ 9657.132176] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [ 9657.137066] Lustre: 375462:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9657.148484] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9657.152295] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9657.207947] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9657.216209] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9659.133263] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9661.658375] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [ 9662.443426] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9662.447062] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9662.458361] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9665.518513] Lustre: 372971:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.206.151@tcp: IR log tst250fs-mdtir failed, not fatal: rc = -2 [ 9672.674799] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9672.676870] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 9672.678905] Lustre: Skipped 3 previous similar messages [ 9672.680432] Lustre: Skipped 2 previous similar messages [ 9677.722887] Lustre: server umount tst250fs-MDT0000 complete [ 9678.939391] LustreError: 373005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784944980 with bad export cookie 5831729830671884618 [ 9678.941658] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9678.943242] LustreError: 373005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9679.063104] Lustre: server umount tst250fs-MDT0001 complete [ 9690.116625] Lustre: server umount tst250fs-OST0000 complete [ 9701.664371] Lustre: server umount tst250fs-OST0001 complete [ 9707.508227] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 9710.023974] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 9719.861449] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 22:03:39 (1784945019) [ 9720.360303] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [ 9725.482425] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 9728.826650] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 9739.422108] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 9510 sec ========== 22:03:59 (1784945039) [ 9740.002028] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 22:03:59 (1784945039) === [ 9741.181698] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 22:04:01 (1784945041) === [ 9754.046523] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 9755.094396] Key type lgssc unregistered [ 9755.225348] LNet: 384574:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9755.227956] LNetError: 384574:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9755.235269] LNet: Removed LNI 192.168.206.151@tcp [ 9755.566114] Key type .llcrypt unregistered [ 9755.567718] Key type ._llcrypt unregistered