[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 496891589 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003054] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.007905] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.008037] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.010132] LSM: Security Framework initializing [ 0.011054] Yama: becoming mindful. [ 0.012037] SELinux: Initializing. [ 0.013074] *** VALIDATE selinux *** [ 0.022526] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027410] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028165] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030114] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031114] *** VALIDATE tmpfs *** [ 0.032494] *** VALIDATE proc *** [ 0.033284] *** VALIDATE cgroup *** [ 0.034014] *** VALIDATE cgroup2 *** [ 0.036243] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037158] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039005] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040033] Spectre V2 : User space: Vulnerable [ 0.042005] Speculative Store Bypass: Vulnerable [ 0.045669] debug: unmapping init [mem 0xffffffff96059000-0xffffffff96060fff] [ 0.047188] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048747] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049024] ... version: 2 [ 0.050016] ... bit width: 48 [ 0.051016] ... generic registers: 4 [ 0.052013] ... value mask: 0000ffffffffffff [ 0.053013] ... max period: 00007fffffffffff [ 0.054017] ... fixed-purpose events: 3 [ 0.055017] ... event mask: 000000070000000f [ 0.056337] rcu: Hierarchical SRCU implementation. [ 0.058374] smp: Bringing up secondary CPUs ... [ 0.059656] x86: Booting SMP configuration: [ 0.060029] .... node #0, CPUs: #1 #2 #3 [ 0.064266] smp: Brought up 1 node, 4 CPUs [ 0.066022] smpboot: Max logical packages: 1 [ 0.067012] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.260144] node 0 deferred pages initialised in 190ms [ 0.264015] devtmpfs: initialized [ 0.265442] x86/mm: Memory block size: 128MB [ 0.267793] gcov: version magic: 0x41383552 [ 0.269300] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.270205] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.271279] pinctrl core: initialized pinctrl subsystem [ 0.272231] [ 0.272868] ************************************************************* [ 0.273011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.274015] ** ** [ 0.275018] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.276016] ** ** [ 0.277014] ** This means that this kernel is built to expose internal ** [ 0.278017] ** IOMMU data structures, which may compromise security on ** [ 0.279013] ** your system. ** [ 0.280016] ** ** [ 0.281017] ** If you see this message and you are not debugging the ** [ 0.282016] ** kernel, report this immediately to your vendor! ** [ 0.283017] ** ** [ 0.284015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.285030] ************************************************************* [ 0.286871] NET: Registered protocol family 16 [ 0.287585] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.288104] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.289098] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.290652] cpuidle: using governor menu [ 0.293501] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.296634] PCI: Using configuration type 1 for base access [ 0.298176] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.308128] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.309051] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.311068] cryptd: max_cpu_qlen set to 1000 [ 0.313343] ACPI: Added _OSI(Module Device) [ 0.314020] ACPI: Added _OSI(Processor Device) [ 0.315021] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.316020] ACPI: Added _OSI(Processor Aggregator Device) [ 0.320113] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.324340] ACPI: Interpreter enabled [ 0.325119] ACPI: PM: (supports S0 S3 S4 S5) [ 0.326023] ACPI: Using IOAPIC for interrupt routing [ 0.328945] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.332514] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.343282] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.346110] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.349022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.351164] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.356470] acpiphp: Slot [2] registered [ 0.358137] acpiphp: Slot [5] registered [ 0.359186] acpiphp: Slot [6] registered [ 0.361180] acpiphp: Slot [7] registered [ 0.362185] acpiphp: Slot [8] registered [ 0.364122] acpiphp: Slot [9] registered [ 0.365176] acpiphp: Slot [10] registered [ 0.366131] acpiphp: Slot [3] registered [ 0.368081] acpiphp: Slot [4] registered [ 0.369164] acpiphp: Slot [11] registered [ 0.371161] acpiphp: Slot [12] registered [ 0.372129] acpiphp: Slot [13] registered [ 0.374207] acpiphp: Slot [14] registered [ 0.376114] acpiphp: Slot [15] registered [ 0.377145] acpiphp: Slot [16] registered [ 0.379152] acpiphp: Slot [17] registered [ 0.380098] acpiphp: Slot [18] registered [ 0.382105] acpiphp: Slot [19] registered [ 0.383108] acpiphp: Slot [20] registered [ 0.384112] acpiphp: Slot [21] registered [ 0.386133] acpiphp: Slot [22] registered [ 0.388152] acpiphp: Slot [23] registered [ 0.389115] acpiphp: Slot [24] registered [ 0.392156] acpiphp: Slot [25] registered [ 0.393137] acpiphp: Slot [26] registered [ 0.395149] acpiphp: Slot [27] registered [ 0.397088] acpiphp: Slot [28] registered [ 0.398109] acpiphp: Slot [29] registered [ 0.399091] acpiphp: Slot [30] registered [ 0.400091] acpiphp: Slot [31] registered [ 0.402092] PCI host bridge to bus 0000:00 [ 0.403000] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.404037] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.406038] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.409039] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.412032] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.415039] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.417224] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.420297] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.424576] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.434918] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.439048] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.441026] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.443028] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.446026] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.450331] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.452798] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.455061] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.460071] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.466025] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.479028] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.485026] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.493033] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.500017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.504029] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.522021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.532926] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.540028] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.548026] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.567029] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.582407] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.590018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.603015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.637022] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.651093] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.656016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.665017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.685017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.696836] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.706019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.712017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.733025] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.745209] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.755018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.765016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.787029] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.801000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.806130] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.809505] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.812521] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.814323] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.821045] iommu: Default domain type: Passthrough [ 0.823672] SCSI subsystem initialized [ 0.825169] ACPI: bus type USB registered [ 0.827169] usbcore: registered new interface driver usbfs [ 0.829132] usbcore: registered new interface driver hub [ 0.832170] usbcore: registered new device driver usb [ 0.834205] pps_core: LinuxPPS API ver. 1 registered [ 0.838021] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.841133] PTP clock support registered [ 0.844189] EDAC MC: Ver: 3.0.0 [ 0.846266] PCI: Using ACPI for IRQ routing [ 0.848150] NetLabel: Initializing [ 0.849022] NetLabel: domain hash size = 128 [ 0.850026] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.851174] NetLabel: unlabeled traffic allowed by default [ 0.854081] vgaarb: loaded [ 0.856414] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.858042] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.864261] clocksource: Switched to clocksource kvm-clock [ 0.980816] VFS: Disk quotas dquot_6.6.0 [ 0.983281] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.986160] *** VALIDATE ramfs *** [ 0.987480] *** VALIDATE hugetlbfs *** [ 0.989331] pnp: PnP ACPI init [ 0.992171] pnp: PnP ACPI: found 6 devices [ 1.023377] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.026789] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.028523] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.030063] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.032048] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.034313] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.036980] NET: Registered protocol family 2 [ 1.039317] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.043712] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.046736] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.051261] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.054041] TCP: Hash tables configured (established 65536 bind 65536) [ 1.056439] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.059618] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.062495] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.065463] NET: Registered protocol family 1 [ 1.067948] RPC: Registered named UNIX socket transport module. [ 1.070343] RPC: Registered udp transport module. [ 1.072349] RPC: Registered tcp transport module. [ 1.074197] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.076935] NET: Registered protocol family 44 [ 1.079070] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.081591] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.083855] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.086400] PCI: CLS 0 bytes, default 64 [ 1.089274] Unpacking initramfs... [ 2.551604] debug: unmapping init [mem 0xffff9ebc7cc54000-0xffff9ebc7ffbffff] [ 2.558223] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.560792] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.565669] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 3.065960] Initialise system trusted keyrings [ 3.068053] Key type blacklist registered [ 3.069930] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.080171] zbud: loaded [ 3.085284] *** VALIDATE nfs *** [ 3.086906] *** VALIDATE nfs4 *** [ 3.088376] pstore: using deflate compression [ 3.092185] Platform Keyring initialized [ 3.205969] NET: Registered protocol family 38 [ 3.207840] Key type asymmetric registered [ 3.209356] Asymmetric key parser 'x509' registered [ 3.211275] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.214415] io scheduler mq-deadline registered [ 3.216518] io scheduler kyber registered [ 3.218249] io scheduler bfq registered [ 3.220263] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.222853] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.225704] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.229180] ACPI: Power Button [PWRF] [ 3.235000] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.242528] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.270096] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.278507] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.300568] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.328532] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.358097] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.363987] Non-volatile memory driver v1.3 [ 3.365723] Linux agpgart interface v0.103 [ 3.400477] virtio_blk virtio1: [vda] 145904 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.405374] vda: detected capacity change from 0 to 74702848 [ 3.420954] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.424489] vdb: detected capacity change from 0 to 1073741824 [ 3.440308] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.443832] vdc: detected capacity change from 0 to 2621440000 [ 3.460818] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.463822] vdd: detected capacity change from 0 to 2621440000 [ 3.480041] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.483602] vde: detected capacity change from 0 to 4294967296 [ 3.498751] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.502586] vdf: detected capacity change from 0 to 4294967296 [ 3.510377] libphy: Fixed MDIO Bus: probed [ 3.515991] usbcore: registered new interface driver usbserial_generic [ 3.519000] usbserial: USB Serial support registered for generic [ 3.521664] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.526473] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.528334] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.531587] mousedev: PS/2 mouse device common for all mice [ 3.535228] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.536574] rtc_cmos 00:05: RTC can wake from S4 [ 3.542649] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.545429] rtc_cmos 00:05: registered as rtc0 [ 3.548250] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.549274] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.551309] intel_pstate: CPU model not supported [ 3.558264] hid: raw HID events driver (C) Jiri Kosina [ 3.560105] usbcore: registered new interface driver usbhid [ 3.561752] usbhid: USB HID core driver [ 3.562913] drop_monitor: Initializing network drop monitor service [ 3.565133] Initializing XFRM netlink socket [ 3.567038] NET: Registered protocol family 10 [ 3.570192] Segment Routing with IPv6 [ 3.571461] NET: Registered protocol family 17 [ 3.573836] mpls_gso: MPLS GSO support [ 3.580234] RAS: Correctable Errors collector initialized. [ 3.582428] AVX version of gcm_enc/dec engaged. [ 3.584172] AES CTR mode by8 optimization enabled [ 3.667480] sched_clock: Marking stable (3667456135, 0)->(4656582489, -989126354) [ 3.671453] registered taskstats version 1 [ 3.673761] Loading compiled-in X.509 certificates [ 3.675933] zswap: loaded using pool lzo/zbud [ 3.703369] Key type big_key registered [ 3.716933] Key type encrypted registered [ 3.718797] ima: No TPM chip found, activating TPM-bypass! [ 3.721044] ima: Allocated hash algorithm: sha1 [ 3.722901] ima: No architecture policies found [ 3.724746] evm: Initialising EVM extended attributes: [ 3.726791] evm: security.selinux [ 3.728051] evm: security.ima [ 3.729579] evm: security.capability [ 3.730982] evm: HMAC attrs: 0x1 [ 3.733660] rtc_cmos 00:05: setting system clock to 2026-08-19 10:07:54 UTC (1787134074) [ 3.739637] debug: unmapping init [mem 0xffffffff97003000-0xffffffff971fffff] [ 3.743341] debug: unmapping init [mem 0xffffffff95d82000-0xffffffff96058fff] [ 3.756087] Write protecting the kernel read-only data: 28672k [ 3.759846] debug: unmapping init [mem 0xffffffff94403000-0xffffffff945fffff] [ 3.763154] debug: unmapping init [mem 0xffffffff94d14000-0xffffffff94dfffff] [ 3.799582] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.808429] systemd[1]: Detected virtualization kvm. [ 3.810575] systemd[1]: Detected architecture x86-64. [ 3.812453] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.844789] systemd[1]: No hostname configured. [ 3.847058] systemd[1]: Set hostname to . [ 3.849363] random: systemd: uninitialized urandom read (16 bytes read) [ 3.852123] systemd[1]: Initializing machine ID from random generator. [ 3.895831] random: ln: uninitialized urandom read (6 bytes read) [ 3.986713] random: systemd: uninitialized urandom read (16 bytes read) [ 3.989816] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.997209] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.002688] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Paths. Starting Create Volatile Files and Directories... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Setup Virtual Console... Starting Journal Service... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.617392] device-mapper: uevent: version 1.0.3 [ 4.620600] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.293309] virtio_net virtio0 ens2: renamed from eth0 [ 5.322278] random: fast init done [ 5.326686] scsi host0: ata_piix [ 5.355288] scsi host1: ata_piix [ 5.356920] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.359907] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.975054] dracut-initqueue[579]: RTNETLINK answers: File exists [ 10.237725] random: crng init done [ 10.239120] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.554725] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.717066] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.990798] SELinux: Disabled at runtime. [ 12.051728] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.061522] systemd[1]: Detected virtualization kvm. [ 12.063550] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.584447] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.588220] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.593937] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.599302] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.603776] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.611920] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.616240] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice User and Session Slice. [ OK ] Listening on Process Core Dump Socket. Mounting Huge Pages File System... [ OK ] Listening on udev Kernel Socket. Mounting POSIX Message Queue File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Starting Remount Root a[ 12.730493] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS nd Kernel File Systems... [ OK ] Reached target Slices. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Mounting Kernel Debug File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Apply Kernel Variables... [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-getty.slice. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.063524] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.388783] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.436274] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.576502] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.589219] EDAC sbridge: Ver: 1.1.2 [ 15.143711] Key type dns_resolver registered [ 15.448182] NFS: Registering the id_resolver key type [ 15.450474] Key type id_resolver registered [ 15.452215] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started daily update of the root trust anchor for DNSSEC. Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg460-server login: [ 42.192082] libcfs: loading out-of-tree module taints kernel. [ 42.213648] Key type ._llcrypt registered [ 42.215183] Key type .llcrypt registered [ 42.267917] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [ 56.927017] hrtimer: interrupt took 3978398 ns [ 58.118928] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 59.256760] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 59.267845] alg: No test for adler32 (adler32-zlib) [ 60.634871] Lustre: Lustre: Build Version: 2.17.57_1_g9da61ce [ 61.573701] LNet: Added LNI 192.168.204.160@tcp [8/256/0/180] [ 63.335401] Key type lgssc registered [ 64.782973] Lustre: Echo OBD driver; http://www.lustre.org/ [ 83.871275] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 120.355343] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 132.774581] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 132.826217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 134.218359] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 134.245384] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 134.307395] Lustre: lustre-MDT0000: new disk, initializing [ 134.362731] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 134.382337] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 139.511693] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 153.529880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 153.648016] Lustre: 6502:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 153.685378] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 153.690703] Lustre: Skipped 1 previous similar message [ 153.783211] Lustre: lustre-MDT0001: new disk, initializing [ 153.856968] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 153.895141] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 153.909560] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 157.971643] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 162.413410] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 171.157325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 171.346324] Lustre: lustre-OST0000: new disk, initializing [ 171.351516] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 171.359929] Lustre: 8442:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 171.422311] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 171.959669] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 171.988494] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 172.062510] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 176.510941] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 191.758028] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 191.905509] Lustre: lustre-OST0001: new disk, initializing [ 191.915375] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 191.923508] Lustre: 9514:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 192.055033] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 197.760401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 197.774622] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 197.947096] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 201.667523] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 217.044620] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 224.403154] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 230.865392] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing check_logdir /tmp/testlogs/ [ 235.898326] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing yml_node [ 240.621169] Lustre: DEBUG MARKER: Client: 2.17.57.1 [ 243.129528] Lustre: DEBUG MARKER: MDS: 2.17.57.1 [ 245.724283] Lustre: DEBUG MARKER: OSS: 2.17.57.1 [ 247.420077] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Aug 19 06:11:56 EDT 2026 [ 265.518040] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 267.144901] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 285.161951] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 285.164596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 285.183310] Lustre: Skipped 1 previous similar message [ 290.275096] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 290.279612] Lustre: Skipped 6 previous similar messages [ 290.398892] Lustre: server umount lustre-MDT0000 complete [ 293.936371] LustreError: 9513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787134364 with bad export cookie 4582067705597388299 [ 293.938437] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 293.946254] LustreError: 9513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 295.395404] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 295.408337] Lustre: Skipped 2 previous similar messages [ 295.417116] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 295.422574] Lustre: Skipped 1 previous similar message [ 298.401477] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 298.416168] Lustre: Skipped 1 previous similar message [ 300.588723] Lustre: server umount lustre-MDT0001 complete [ 304.494390] Lustre: server umount lustre-OST0000 complete [ 308.947245] Lustre: server umount lustre-OST0001 complete [ 317.618338] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [ 329.117634] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 367.314561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 367.493581] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 367.530549] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 367.579856] Lustre: lustre-MDT0000: new disk, initializing [ 367.631813] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 367.650885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 373.467168] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 385.380706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 385.543259] Lustre: 15069:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 385.605720] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 385.611490] Lustre: Skipped 1 previous similar message [ 385.829707] Lustre: lustre-MDT0001: new disk, initializing [ 386.048833] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 386.097544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 386.123406] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 392.185903] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 404.159760] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 411.803424] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 419.544602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 419.866862] Lustre: lustre-OST0000: new disk, initializing [ 419.871351] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 419.882668] Lustre: 17042:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 420.022063] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 421.392220] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 421.404432] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 421.503801] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 427.745459] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 438.318432] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 443.962813] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 444.210662] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 449.549077] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 449.746669] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 452.069650] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 452.072812] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 452.083731] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 452.084231] LustreError: Skipped 1 previous similar message [ 452.128496] Lustre: Skipped 2 previous similar messages [ 462.319969] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 462.326827] Lustre: Skipped 4 previous similar messages [ 466.399205] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 466.639919] Lustre: server umount lustre-OST0000 complete [ 472.544915] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 472.561084] Lustre: Skipped 1 previous similar message [ 477.982866] Lustre: server umount lustre-MDT0000 complete [ 482.788640] LustreError: 15081:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 482.815128] LustreError: 15081:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 482.851241] LustreError: 15062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787134553 with bad export cookie 4582067705597389832 [ 482.864805] LustreError: 15062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 482.866107] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 483.958442] Lustre: server umount lustre-MDT0001 complete [ 491.001780] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 06:15:59 (1787134559) [ 511.662794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 512.140283] LustreError: 19920:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 512.230627] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 517.605229] LustreError: 19921:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 518.185835] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 521.700867] LustreError: 19920:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 526.817042] LustreError: 19921:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 528.702191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 534.146810] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 542.313409] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 549.164708] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 551.634375] Lustre: Failing over lustre-MDT0000 [ 551.944096] Lustre: server umount lustre-MDT0000 complete [ 553.698519] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 553.733190] LustreError: 19916:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 553.754567] LustreError: 19916:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 569.313953] Lustre: 3640:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787134624/real 1787134624] req@ffff9ebbc33e2680 x1873945959758848/t0(0) o400->MGC192.168.204.160@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787134640 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 569.340787] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 572.707639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 573.473415] LustreError: 20649:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 573.500498] LustreError: 20649:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 580.576984] LustreError: 3636:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9ebcffa2c380 x1873945959761792/t0(0) o250->MGC192.168.204.160@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 580.886879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 580.896549] Lustre: Skipped 1 previous similar message [ 585.878595] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 586.212961] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 586.235217] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 586.252382] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 596.510706] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 601.568601] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 601.583736] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 601.606513] Lustre: Skipped 1 previous similar message [ 601.615586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 601.625546] Lustre: Skipped 4 previous similar messages [ 606.865987] Lustre: server umount lustre-MDT0000 complete [ 608.736484] LustreError: 20649:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 608.779625] LustreError: 20649:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 611.209114] LustreError: 20659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787134681 with bad export cookie 4582067705597391246 [ 611.222787] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 611.227091] LustreError: 20659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 611.638358] Lustre: server umount lustre-MDT0001 complete [ 620.801932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 626.219741] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 635.093625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 639.904781] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 648.061757] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 654.948516] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 657.595665] Lustre: Failing over lustre-MDT0000 [ 657.835981] Lustre: server umount lustre-MDT0000 complete [ 660.706652] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 660.721752] Lustre: Skipped 1 previous similar message [ 676.320251] LustreError: 24504:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 676.322197] Lustre: 3638:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787134731/real 1787134731] req@ffff9ebcfd8b4380 x1873945959789568/t0(0) o400->MGC192.168.204.160@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787134747 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 676.339719] LustreError: 24504:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 676.392795] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 678.842057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 687.954869] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 693.011643] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 693.227847] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 693.248515] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 693.258928] Lustre: Skipped 1 previous similar message [ 693.281674] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 703.485274] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 708.578070] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 708.591738] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 708.608400] Lustre: Skipped 1 previous similar message [ 708.621021] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 708.627361] Lustre: Skipped 2 previous similar messages [ 722.399492] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 722.639385] Lustre: server umount lustre-MDT0000 complete [ 726.615688] LustreError: 23245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787134797 with bad export cookie 4582067705597392184 [ 726.630265] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 726.631231] LustreError: 23245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 726.882228] Lustre: server umount lustre-MDT0001 complete [ 735.689552] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 06:20:04 (1787134804) [ 753.929150] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [ 768.652822] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 813.569612] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 827.735782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 827.990982] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 828.012591] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 828.084578] Lustre: lustre-MDT0000: new disk, initializing [ 828.168947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 828.178109] Lustre: Skipped 3 previous similar messages [ 828.210471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 833.656397] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 846.428348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 846.554311] Lustre: 29659:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 847.037704] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 852.065378] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 862.669520] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 869.701273] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 877.694483] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 878.105555] Lustre: 31629:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 879.915975] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 880.016316] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 880.100598] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 880.125875] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 885.754656] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 896.784177] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 903.082772] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 931.325644] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 931.329864] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 931.330735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 931.358147] Lustre: Skipped 2 previous similar messages [ 931.396063] Lustre: Skipped 7 previous similar messages [ 936.374041] Lustre: server umount lustre-OST0000 complete [ 948.950441] Lustre: server umount lustre-MDT0000 complete [ 951.778480] LustreError: 29670:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 951.793595] LustreError: 29670:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 953.078780] LustreError: 29652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787135023 with bad export cookie 4582067705597392590 [ 953.090748] LustreError: 29652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 953.095350] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 953.571844] Lustre: server umount lustre-MDT0001 complete [ 966.004361] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [ 968.497887] Key type lgssc unregistered [ 968.780836] LNet: 33618:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 968.790925] LNetError: 33618:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 968.805845] LNet: Removed LNI 192.168.204.160@tcp [ 969.788137] Key type .llcrypt unregistered [ 969.792617] Key type ._llcrypt unregistered [ 996.576778] Key type ._llcrypt registered [ 996.579408] Key type .llcrypt registered [ 996.671887] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 997.961021] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 998.006152] alg: No test for adler32 (adler32-zlib) [ 999.116575] Lustre: Lustre: Build Version: 2.17.57_1_g9da61ce [ 999.462044] LNet: Added LNI 192.168.204.160@tcp [8/256/0/180] [ 1001.143168] Key type lgssc registered [ 1002.394351] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1011.363717] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1011.377641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1012.824548] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1016.984466] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 1022.947714] LustreError: 35031:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1022.963088] LustreError: 35031:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1026.123401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1026.480571] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1031.497542] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 1039.324646] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1045.006452] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1052.132586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1052.512822] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1057.678297] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1058.355717] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 1066.411645] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1071.999441] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1083.009859] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 06:25:52 (1787135152) [ 1084.823701] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1086.670087] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1094.428573] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 06:26:02 (1787135162) [ 1105.062888] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 06:26:13 (1787135173) [ 1106.271835] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1115.208441] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1115.214228] Lustre: Skipped 2 previous similar messages [ 1117.431149] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 06:26:25 (1787135185) [ 1119.486213] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1119.493258] Lustre: Skipped 2 previous similar messages [ 1128.413237] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1128.419957] Lustre: Skipped 4 previous similar messages [ 1145.089402] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 06:26:53 (1787135213) [ 1148.109378] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1152.427688] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1162.193625] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 06:27:10 (1787135230) [ 1164.515993] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1164.522299] Lustre: Skipped 6 previous similar messages [ 1179.722731] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 06:27:28 (1787135248) [ 1196.000133] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1196.007556] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1196.020083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1200.098982] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1200.113705] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1200.119054] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1200.138881] Lustre: Skipped 1 previous similar message [ 1205.219531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1205.236613] Lustre: Skipped 1 previous similar message [ 1210.336358] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1210.355317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1210.360979] Lustre: Skipped 1 previous similar message [ 1210.562715] Lustre: server umount lustre-OST0000 complete [ 1215.471364] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1215.476963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1215.489898] Lustre: Skipped 1 previous similar message [ 1215.504374] Lustre: Skipped 1 previous similar message [ 1225.698919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1225.707245] Lustre: Skipped 3 previous similar messages [ 1229.279189] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1229.451630] Lustre: server umount lustre-MDT0000 complete [ 1230.821099] LustreError: 35031:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1233.749593] LustreError: 36876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787135304 with bad export cookie 12382455281278527328 [ 1233.768963] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1233.772508] LustreError: 36876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1234.280520] Lustre: server umount lustre-MDT0001 complete [ 1247.440844] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [ 1250.145736] Key type lgssc unregistered [ 1250.427641] LNet: 42892:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1250.439734] LNetError: 42892:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1250.463865] LNet: Removed LNI 192.168.204.160@tcp [ 1251.251579] Key type .llcrypt unregistered [ 1251.255534] Key type ._llcrypt unregistered [ 1270.953220] Key type ._llcrypt registered [ 1270.958736] Key type .llcrypt registered [ 1271.087530] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [ 1289.448074] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 1291.063381] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1291.101088] alg: No test for adler32 (adler32-zlib) [ 1292.302536] Lustre: Lustre: Build Version: 2.17.57_1_g9da61ce [ 1292.514963] LNet: Added LNI 192.168.204.160@tcp [8/256/0/180] [ 1294.167189] Key type lgssc registered [ 1295.578551] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1342.170292] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1342.199686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1343.565262] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1343.622928] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1343.739704] Lustre: lustre-MDT0000: new disk, initializing [ 1343.850112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1343.887484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1348.131757] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 1359.897161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1359.993994] Lustre: 46048:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1360.073978] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1360.077148] Lustre: Skipped 1 previous similar message [ 1360.149646] Lustre: lustre-MDT0001: new disk, initializing [ 1360.236399] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1360.272123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1360.290718] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1365.065955] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 1376.321889] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1383.491269] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1391.604159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1391.997413] Lustre: lustre-OST0000: new disk, initializing [ 1392.006495] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1392.016883] Lustre: 48018:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1392.115042] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1398.365967] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1398.398973] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1398.563358] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1399.667359] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 1412.464970] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1419.155030] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1419.511936] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1426.367522] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1426.843404] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1434.600822] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1434.616481] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1434.624528] Lustre: Skipped 1 previous similar message [ 1434.625592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1434.647971] LustreError: Skipped 1 previous similar message [ 1435.922222] Lustre: server umount lustre-OST0000 complete [ 1442.275919] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1442.293308] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1442.316422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1442.321600] Lustre: Skipped 1 previous similar message [ 1444.833022] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1444.835259] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1446.738893] Lustre: server umount lustre-MDT0000 complete [ 1450.620314] LustreError: 46041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787135521 with bad export cookie 13093148995876383243 [ 1450.623890] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1450.637447] LustreError: 46041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1451.048704] Lustre: server umount lustre-MDT0001 complete [ 1463.452945] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 06:32:11 (1787135531) [ 1473.177397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1473.886648] LustreError: 50478:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1473.931424] LustreError: 50478:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1474.052683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1479.140259] LustreError: 50479:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1479.153489] LustreError: 50479:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1480.764391] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 1484.269728] LustreError: 50478:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1489.382746] LustreError: 50479:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1489.973681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1490.522871] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1495.886954] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 1505.165731] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1511.609190] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1519.228786] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1519.515643] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1525.855425] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 1534.876135] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1540.692853] LustreError: 53008:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1541.899838] Lustre: Modifying parameter general.timeout=1 in log params [ 1543.324605] Lustre: Modifying parameter general.timeout=2 in log params [ 1545.728265] Lustre: Modifying parameter general.timeout=4 in log params [ 1545.732547] Lustre: Skipped 1 previous similar message [ 1550.150077] Lustre: Modifying parameter general.timeout=8 in log params [ 1550.156872] Lustre: Skipped 3 previous similar messages [ 1558.324257] Lustre: Modifying parameter general.timeout=16 in log params [ 1558.328499] Lustre: Skipped 7 previous similar messages [ 1574.337070] Lustre: Modifying parameter general.timeout=32 in log params [ 1574.343341] Lustre: Skipped 15 previous similar messages [ 1606.843392] Lustre: Modifying parameter general.timeout=64 in log params [ 1606.851594] Lustre: Skipped 31 previous similar messages [ 1671.614398] Lustre: Modifying parameter general.timeout=129 in log params [ 1671.618480] Lustre: Skipped 64 previous similar messages [ 1684.344328] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 06:35:52 (1787135752) [ 1687.897179] Lustre: 59962:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1690.379074] Lustre: 60062:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1692.696746] Lustre: 60173:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1696.847238] Lustre: 60369:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1696.856274] Lustre: 60369:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1726.709251] Lustre: 61844:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1726.718464] Lustre: 61844:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1726.746229] LustreError: 61844:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1730.508722] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 06:36:39 (1787135799) [ 1743.333462] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1743.335842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1743.354959] Lustre: Skipped 2 previous similar messages [ 1743.366107] Lustre: Skipped 3 previous similar messages [ 1748.449233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1748.457919] Lustre: Skipped 2 previous similar messages [ 1748.718709] Lustre: server umount lustre-MDT0000 complete [ 1753.189597] LustreError: 50458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787135823 with bad export cookie 13093148995876384125 [ 1753.200383] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1753.208056] LustreError: 50458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1753.553366] Lustre: server umount lustre-MDT0001 complete [ 1768.178734] Lustre: server umount lustre-OST0000 complete [ 1797.287260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1797.564740] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1797.748159] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1802.317543] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 1811.559226] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1811.649505] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1811.670523] Lustre: 64374:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1816.334958] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 1824.483410] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1831.005354] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1837.810929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1837.978792] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1838.239301] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1838.246047] Lustre: Skipped 1 previous similar message [ 1844.603373] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 1853.214459] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1854.934335] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1854.949373] Lustre: Skipped 24 previous similar messages [ 1854.998285] Lustre: 66940:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1865.189771] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1865.199676] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1865.219400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1867.289931] Lustre: server umount lustre-OST0000 complete [ 1873.377969] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1873.388355] LustreError: Skipped 1 previous similar message [ 1873.391849] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1873.404515] Lustre: Skipped 1 previous similar message [ 1885.151459] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1886.178601] LustreError: 64380:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1886.195199] LustreError: 64380:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1887.331945] Lustre: server umount lustre-MDT0000 complete [ 1891.313044] LustreError: 64367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787135962 with bad export cookie 13093148995876387317 [ 1891.322762] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1891.325132] LustreError: 64367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1891.649696] Lustre: server umount lustre-MDT0001 complete [ 1903.448240] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [ 1906.278706] Key type lgssc unregistered [ 1906.543811] LNet: 68153:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1906.550442] LNetError: 68153:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1906.571101] LNet: Removed LNI 192.168.204.160@tcp [ 1907.494044] Key type .llcrypt unregistered [ 1907.501745] Key type ._llcrypt unregistered [ 1932.231297] Key type ._llcrypt registered [ 1932.234340] Key type .llcrypt registered [ 1932.351783] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 1933.488510] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1933.502117] alg: No test for adler32 (adler32-zlib) [ 1934.687380] Lustre: Lustre: Build Version: 2.17.57_1_g9da61ce [ 1935.051467] LNet: Added LNI 192.168.204.160@tcp [8/256/0/180] [ 1936.711189] Key type lgssc registered [ 1938.561611] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1947.778582] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1947.809221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1949.522795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1954.176204] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 1959.905777] LustreError: 69568:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1959.939925] LustreError: 69568:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1964.502595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1965.125977] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1969.837845] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 1977.432052] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1983.652299] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1993.197941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1993.755312] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2003.225361] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 2011.394866] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2021.360711] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2036.196665] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2036.204454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2036.221164] Lustre: Skipped 2 previous similar messages [ 2036.234652] Lustre: Skipped 2 previous similar messages [ 2038.250474] Lustre: server umount lustre-MDT0000 complete [ 2041.312856] LustreError: 69568:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2041.351627] LustreError: 69568:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2042.618442] LustreError: 72527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787136113 with bad export cookie 57648468272824272 [ 2042.621793] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2042.636860] LustreError: 72527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2043.179791] Lustre: server umount lustre-MDT0001 complete [ 2058.204728] Lustre: server umount lustre-OST0000 complete [ 2090.671287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2090.928938] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2090.965952] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2091.330718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2096.391035] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 2106.655730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2106.812958] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2106.847208] Lustre: 74442:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2107.454233] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2111.933993] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 2120.581559] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2127.021759] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2133.642497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2133.842614] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2134.143717] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2140.090182] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 2148.237879] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2150.025920] Lustre: 77017:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2151.990083] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2158.560588] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2158.566037] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2158.582953] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2159.092622] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2163.366831] Lustre: server umount lustre-OST0000 complete [ 2168.804249] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2168.815643] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2168.825229] Lustre: Skipped 1 previous similar message [ 2168.830240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2168.837028] Lustre: Skipped 2 previous similar messages [ 2173.526878] Lustre: server umount lustre-MDT0000 complete [ 2174.439747] LustreError: 75472:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2174.471994] LustreError: 75472:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2177.469724] LustreError: 74433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787136248 with bad export cookie 57648468272825119 [ 2177.473799] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2177.479044] LustreError: 74433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2177.830589] Lustre: server umount lustre-MDT0001 complete [ 2189.813489] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [ 2193.170734] Key type lgssc unregistered [ 2193.558574] LNet: 78232:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2193.573171] LNetError: 78232:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2193.599303] LNet: Removed LNI 192.168.204.160@tcp [ 2194.725710] Key type .llcrypt unregistered [ 2194.727465] Key type ._llcrypt unregistered [ 2222.234309] Key type ._llcrypt registered [ 2222.237284] Key type .llcrypt registered [ 2222.345799] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 2223.653498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2223.683225] alg: No test for adler32 (adler32-zlib) [ 2224.762499] Lustre: Lustre: Build Version: 2.17.57_1_g9da61ce [ 2224.965189] LNet: Added LNI 192.168.204.160@tcp [8/256/0/180] [ 2226.611403] Key type lgssc registered [ 2227.809477] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2235.034937] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2235.052202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2236.522063] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2240.643364] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 2247.136905] LustreError: 79645:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2247.149386] LustreError: 79645:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2249.250781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2249.798707] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2255.019702] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 2264.904574] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2271.556826] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2279.239791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2280.066148] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2288.758602] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 2297.658891] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2305.100277] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2306.339400] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2307.463247] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2309.745251] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2309.751867] Lustre: Skipped 1 previous similar message [ 2314.305941] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2314.310712] Lustre: Skipped 4 previous similar messages [ 2322.760958] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2322.764304] Lustre: Skipped 8 previous similar messages [ 2339.713078] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2339.718210] Lustre: Skipped 17 previous similar messages [ 2371.967285] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2371.971651] Lustre: Skipped 33 previous similar messages [ 2413.508723] Lustre: 87749:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2417.442309] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 06:48:05 (1787136485) [ 2419.565807] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2421.466465] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 06:48:10 (1787136490) [ 2427.882989] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2427.893486] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2427.912061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2430.948279] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2430.950741] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2430.965229] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2430.994819] Lustre: Skipped 1 previous similar message [ 2432.377403] Lustre: server umount lustre-OST0000 complete [ 2439.142726] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2439.154697] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2439.183457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2441.185048] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2441.186310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2443.219242] Lustre: server umount lustre-MDT0000 complete [ 2446.306971] LustreError: 79646:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2446.317298] LustreError: 79646:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2447.319304] LustreError: 79627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787136518 with bad export cookie 15766599115216086337 [ 2447.322360] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2447.335162] LustreError: 79627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2447.654930] Lustre: server umount lustre-MDT0001 complete [ 2460.128425] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [ 2463.304683] Key type lgssc unregistered [ 2463.746911] LNet: 89004:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2463.762960] LNetError: 89004:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2464.805828] LNet: Removed LNI 192.168.204.160@tcp [ 2465.905318] Key type .llcrypt unregistered [ 2465.909629] Key type ._llcrypt unregistered [ 2471.526392] Key type ._llcrypt registered [ 2471.528711] Key type .llcrypt registered [ 2471.690298] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2476.913465] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules [ 2478.066541] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2478.111337] alg: No test for adler32 (adler32-zlib) [ 2479.453591] Lustre: Lustre: Build Version: 2.17.57_1_g9da61ce [ 2479.802228] LNet: Added LNI 192.168.204.160@tcp [8/256/0/180] [ 2481.537115] Key type lgssc registered [ 2483.570805] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2485.359093] LustreError: 90233:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2506.857083] LustreError: 90233:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2507.545751] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 2519.819870] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2519.852287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2521.769091] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2526.879886] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 2531.298543] LustreError: 91233:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2531.318466] LustreError: 91233:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2536.423358] LustreError: 91232:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2537.009516] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 06:50:05 (1787136605) [ 2541.570478] LustreError: 91233:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2542.988697] LustreError: 91265:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2543.003335] LustreError: 91265:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2543.020683] LustreError: 91265:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 2543.526942] Lustre: server umount lustre-MDT0000 complete [ 2561.250247] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [ 2564.515831] Key type lgssc unregistered [ 2564.787091] LNet: 92763:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2564.801338] LNetError: 92763:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2564.819043] LNet: Removed LNI 192.168.204.160@tcp [ 2565.946649] Key type .llcrypt unregistered [ 2565.950159] Key type ._llcrypt unregistered [ 2588.629699] Key type ._llcrypt registered [ 2588.632027] Key type .llcrypt registered [ 2588.732030] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 2589.829333] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2589.913885] alg: No test for adler32 (adler32-zlib) [ 2590.953861] Lustre: Lustre: Build Version: 2.17.57_1_g9da61ce [ 2591.393981] LNet: Added LNI 192.168.204.160@tcp [8/256/0/180] [ 2593.152519] Key type lgssc registered [ 2594.432768] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2604.396203] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2604.440754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2606.129843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2611.581798] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 2615.284536] LustreError: 93986:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2615.309330] LustreError: 93986:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2620.385357] LustreError: 93987:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2621.870305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2622.808112] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2627.866328] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 2636.052264] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2641.933599] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2650.260342] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2650.611407] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 2650.978933] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2658.204711] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 2666.577941] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2725.033459] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 06:53:13 (1787136793) [ 2727.483749] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2730.167894] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 06:53:18 (1787136798) [ 2735.079087] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2735.089889] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2735.101331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2739.172980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2739.176256] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2741.274367] Lustre: server umount lustre-MDT0000 complete [ 2744.288593] LustreError: 93986:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2744.333337] LustreError: 93986:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2747.807109] LustreError: 93966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787136818 with bad export cookie 16222255805482084223 [ 2747.812702] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2747.836598] LustreError: 93966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2748.595067] Lustre: server umount lustre-MDT0001 complete [ 2765.960977] Lustre: server umount lustre-OST0000 complete [ 2779.187824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2779.750944] LustreError: 98284:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2779.919349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2785.239573] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 2785.263228] LustreError: 98285:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2794.468709] LustreError: 98285:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2794.501736] LustreError: 98285:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2796.554661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2797.349736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2802.970685] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 2812.451899] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2819.530479] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2834.725381] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2834.981293] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2834.986821] LustreError: 98274:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2835.025688] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2835.038992] LustreError: 100369:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2835.047638] LustreError: 100369:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2835.057107] LustreError: 100369:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2835.163681] Lustre: server umount lustre-OST0000 complete [ 2835.168475] LustreError: 100369:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2844.567311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2844.797449] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2844.805116] LustreError: 98274:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2844.830143] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2844.842427] LustreError: 100836:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2844.855523] LustreError: 100836:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2844.864539] LustreError: 100836:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2844.951901] Lustre: server umount lustre-OST0000 complete [ 2844.954142] LustreError: 100836:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2855.534701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2855.722435] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2855.726478] Lustre: Found index 0 for lustre-OST0000, updating log [ 2855.735754] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2855.810640] Lustre: lustre-OST0000: new disk, initializing [ 2855.813435] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2855.821813] Lustre: 101370:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2856.044227] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2863.181725] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 2867.581630] Lustre: Failing over lustre-OST0000 [ 2867.897834] Lustre: server umount lustre-OST0000 complete [ 2873.835328] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2873.848067] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2873.873372] Lustre: Skipped 1 previous similar message [ 2873.879589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2873.885493] Lustre: Skipped 2 previous similar messages [ 2876.897028] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2878.411330] Lustre: server umount lustre-MDT0000 complete [ 2882.017406] LustreError: 98285:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2882.033032] LustreError: 98285:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2882.426193] LustreError: 98266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787136953 with bad export cookie 16222255805482086974 [ 2882.428739] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2882.437690] LustreError: 98266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2892.968127] Lustre: server umount lustre-MDT0001 complete [ 2902.736841] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 06:56:11 (1787136971) [ 2920.729735] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 2931.185426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2931.475958] LustreError: 103721:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2931.615112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2935.690519] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 2944.879342] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2950.549735] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 2953.312661] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2960.797953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2961.373548] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2969.101359] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 2969.513289] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2969.558867] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2969.566479] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2979.198330] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2979.299258] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2979.384568] Lustre: lustre-OST0001: new disk, initializing [ 2979.396562] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2979.412648] Lustre: 106042:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2979.816954] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2979.822844] Lustre: Skipped 2 previous similar messages [ 2986.605547] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2986.614791] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2986.733412] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2988.128777] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 2998.162361] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3007.259296] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3012.583170] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3012.601553] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 3012.603875] Lustre: Skipped 2 previous similar messages [ 3012.604131] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3012.604135] Lustre: Skipped 2 previous similar messages [ 3018.712766] Lustre: server umount lustre-MDT0001 complete [ 3022.050594] LustreError: 105236:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3022.082614] LustreError: 105236:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3030.446320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3030.544581] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3030.549652] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3030.595728] Lustre: 103710:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3032.276636] LustreError: 107581:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3032.289375] LustreError: 107581:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3032.313900] LustreError: 103713:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3032.446448] LustreError: 107589:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.160@tcp: cfg command failed: rc = -17 [ 3032.458142] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3032.458142] [ 3036.671755] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3037.935676] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 3053.300245] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 06:58:42 (1787137122) [ 3074.989201] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 3095.246721] Lustre: 109659:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3119.970177] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3123.992442] Lustre: 110794:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3215.327835] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3215.340305] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3215.369296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3215.379795] Lustre: Skipped 6 previous similar messages [ 3220.962986] LustreError: 105236:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3220.981897] LustreError: 105236:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 3221.183405] Lustre: server umount lustre-MDT0000 complete [ 3226.051145] LustreError: 104861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787137296 with bad export cookie 16222255805482087975 [ 3226.056368] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3226.061313] LustreError: 104861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3226.369534] Lustre: server umount lustre-MDT0001 complete [ 3233.366738] Lustre: server umount lustre-OST0000 complete [ 3237.875013] Lustre: server umount lustre-OST0001 complete [ 3244.030621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3254.667334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3263.635779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3275.372784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3296.311282] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 3306.862071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3306.886184] Lustre: lustre-MDT0000: reset Object Index mappings [ 3307.504677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3307.513638] Lustre: Skipped 1 previous similar message [ 3311.935479] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 3320.781326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3320.800649] Lustre: lustre-MDT0001: reset Object Index mappings [ 3325.580131] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 3328.784402] Lustre: 115908:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3335.964192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3343.001060] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 3346.039443] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:232 to 0x280000400:257) [ 3351.811355] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3352.570185] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:349 to 0x280000401:386) [ 3358.195928] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3358.205924] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3359.722250] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 3368.351929] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3373.078827] Lustre: 117782:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3439.669637] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 07:05:08 (1787137508) [ 3469.792280] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3469.804616] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3469.825929] Lustre: Skipped 5 previous similar messages [ 3469.836466] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3469.840598] Lustre: Skipped 5 previous similar messages [ 3472.351635] Lustre: server umount lustre-MDT0000 complete [ 3475.690238] LustreError: 114738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787137546 with bad export cookie 16222255805482134868 [ 3475.696376] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3475.705207] LustreError: 114738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3481.058419] LustreError: 114753:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3481.095095] LustreError: 114753:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 3482.285783] Lustre: server umount lustre-MDT0001 complete [ 3493.360759] Lustre: server umount lustre-OST0000 complete [ 3511.263124] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3511.506366] Lustre: server umount lustre-OST0001 complete [ 3521.468898] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [ 3533.661386] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 3573.143111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3573.447923] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3573.526113] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3573.594358] Lustre: lustre-MDT0000: new disk, initializing [ 3573.687855] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3573.692257] Lustre: Skipped 3 previous similar messages [ 3573.709303] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3579.207557] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 3586.553444] Lustre: server umount lustre-MDT0000 complete [ 3597.627819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3597.968245] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3597.989919] Lustre: 122059:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3598.014567] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3603.497510] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 3608.780447] Lustre: server umount lustre-MDT0000 complete [ 3618.269749] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 07:08:06 (1787137686) [ 3620.393374] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3622.356515] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 07:08:11 (1787137691) [ 3642.228996] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 3653.694483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3658.605442] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 3667.394512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3667.553589] Lustre: 124107:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3667.600424] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3667.603553] Lustre: Skipped 1 previous similar message [ 3667.670661] Lustre: lustre-MDT0001: new disk, initializing [ 3667.814955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3667.836846] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3672.756261] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 3677.993607] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3684.959098] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3685.286304] Lustre: lustre-OST0000: new disk, initializing [ 3685.290507] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3685.297755] Lustre: 125686:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3686.975781] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3686.996764] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3687.132105] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3692.318677] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 3703.910444] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3704.158864] Lustre: lustre-OST0001: new disk, initializing [ 3704.166953] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3704.183142] Lustre: 126559:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3705.709944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3705.734745] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3705.869597] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3710.992230] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 3721.234207] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3725.467852] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4165.683306] Lustre: 132988:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4165.702395] Lustre: 132988:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4165.719147] Lustre: 132988:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4165.735893] Lustre: 132988:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4165.753370] Lustre: 132988:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4165.775981] Lustre: 132988:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4504.748643] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 07:22:51 (1787138571) [ 4521.786372] Lustre: server umount lustre-MDT0000 complete [ 4523.491310] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4523.506406] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4523.517664] Lustre: Skipped 5 previous similar messages [ 4523.520931] LustreError: 125680:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4525.982047] LustreError: 126558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787138596 with bad export cookie 16222255805482205659 [ 4526.003731] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4526.007962] LustreError: 126558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4526.057493] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4526.078753] Lustre: Skipped 8 previous similar messages [ 4528.607453] Lustre: server umount lustre-MDT0001 complete [ 4539.546611] Lustre: server umount lustre-OST0000 complete [ 4558.303146] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4558.596612] Lustre: server umount lustre-OST0001 complete [ 4566.976524] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [ 4578.435541] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 4621.869169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4622.177212] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4622.273990] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4622.343733] Lustre: lustre-MDT0000: new disk, initializing [ 4622.399533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4622.413925] Lustre: Skipped 5 previous similar messages [ 4622.437252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4627.385288] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 4641.133325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4641.301872] Lustre: 136048:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4641.356880] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4641.362519] Lustre: Skipped 1 previous similar message [ 4641.455526] Lustre: lustre-MDT0001: new disk, initializing [ 4641.614401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4641.627560] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4645.695854] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 4654.725940] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4660.133997] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4666.049682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4666.328477] Lustre: lustre-OST0000: new disk, initializing [ 4666.334391] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4666.339990] Lustre: 138019:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4666.427936] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4666.443620] Lustre: Skipped 1 previous similar message [ 4667.566155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4667.583542] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4667.718484] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4672.318988] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 4682.031407] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4691.729782] Lustre: lustre-MDD0000: changelog on [ 4694.926705] Lustre: lustre-MDD0001: changelog on [ 6838.969645] Lustre: lustre-MDD0001: changelog off [ 6841.572742] Lustre: 136056:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6842.600583] Lustre: lustre-MDD0000: changelog off [ 6854.112569] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6854.120169] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6854.124915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6854.134336] Lustre: Skipped 5 previous similar messages [ 6854.147673] Lustre: Skipped 1 previous similar message [ 6856.030656] Lustre: server umount lustre-OST0000 complete [ 6863.844918] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6863.857401] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6863.881455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6867.088647] Lustre: server umount lustre-MDT0000 complete [ 6869.473236] LustreError: 136058:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6869.497679] LustreError: 136058:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6871.104569] LustreError: 138018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787140941 with bad export cookie 16222255805483026248 [ 6871.106250] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6871.117119] LustreError: 138018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6871.650422] Lustre: server umount lustre-MDT0001 complete [ 6885.746918] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [ 6889.465305] Key type lgssc unregistered [ 6889.743718] LNet: 147208:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6889.760421] LNetError: 147208:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6889.789356] LNet: Removed LNI 192.168.204.160@tcp [ 6890.804201] Key type .llcrypt unregistered [ 6890.807194] Key type ._llcrypt unregistered [ 6914.311619] Key type ._llcrypt registered [ 6914.317254] Key type .llcrypt registered [ 6914.481763] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [ 6938.606537] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 6940.143056] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6940.197597] alg: No test for adler32 (adler32-zlib) [ 6941.314682] Lustre: Lustre: Build Version: 2.17.57_1_g9da61ce [ 6941.582650] LNet: Added LNI 192.168.204.160@tcp [8/256/0/180] [ 6943.319969] Key type lgssc registered [ 6944.642696] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6987.722915] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6987.770324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6989.040487] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6989.058805] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6989.129792] Lustre: lustre-MDT0000: new disk, initializing [ 6989.214446] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6989.233870] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6993.843978] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 7005.483909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7005.652141] Lustre: 150415:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7005.723411] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7005.727259] Lustre: Skipped 1 previous similar message [ 7005.824221] Lustre: lustre-MDT0001: new disk, initializing [ 7005.949444] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7006.002778] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7006.031606] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7011.667925] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 7021.459874] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7027.728621] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7034.990694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7035.306963] Lustre: lustre-OST0000: new disk, initializing [ 7035.314470] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7035.321419] Lustre: 152381:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7035.430445] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7042.085587] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 7045.718876] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7045.732760] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7045.867168] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7054.057984] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7061.376063] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_module obdecho/obdecho [ 7063.450095] LustreError: 153384:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 7063.459346] LustreError: 153384:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 7065.608546] LustreError: 153480:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 7081.956879] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7081.963474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7081.969103] Lustre: Skipped 2 previous similar messages [ 7081.989494] Lustre: Skipped 2 previous similar messages [ 7084.326491] Lustre: server umount lustre-MDT0000 complete [ 7088.366248] LustreError: 153823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787141159 with bad export cookie 17165256071151297613 [ 7088.369766] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7088.377577] LustreError: 153823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7088.790334] Lustre: server umount lustre-MDT0001 complete [ 7103.440532] Lustre: server umount lustre-OST0000 complete [ 7126.392305] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [ 7136.328452] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 7170.966506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7171.295680] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7171.322545] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7171.404808] Lustre: lustre-MDT0000: new disk, initializing [ 7171.505550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7171.521650] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7175.948447] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 7188.107139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7188.302872] Lustre: 157200:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7188.394629] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7188.400448] Lustre: Skipped 1 previous similar message [ 7188.515456] Lustre: lustre-MDT0001: new disk, initializing [ 7188.638579] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7188.723544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7188.740804] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7194.361569] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 7205.284607] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7211.917517] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7218.312528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7218.574906] Lustre: lustre-OST0000: new disk, initializing [ 7218.580715] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7218.587832] Lustre: 159171:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7218.675788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7224.689631] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 7226.421035] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7226.427899] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7226.527850] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7233.890621] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7238.880312] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7239.140350] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7243.731765] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7243.921417] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7247.330158] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7247.336864] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7247.366870] Lustre: Skipped 1 previous similar message [ 7247.370243] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7247.375894] Lustre: Skipped 1 previous similar message [ 7252.281209] Lustre: server umount lustre-OST0000 complete [ 7257.568821] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7257.590138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7257.597302] Lustre: Skipped 1 previous similar message [ 7262.690855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7262.696469] Lustre: Skipped 1 previous similar message [ 7267.822105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7267.829781] Lustre: Skipped 1 previous similar message [ 7270.367122] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7270.526505] Lustre: server umount lustre-MDT0000 complete [ 7272.928641] LustreError: 157211:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7272.952282] LustreError: 157211:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 7274.368274] LustreError: 157192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787141345 with bad export cookie 17165256071151298572 [ 7274.377241] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7274.383889] LustreError: 157192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7274.744660] Lustre: server umount lustre-MDT0001 complete [ 7278.817627] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 08:09:07 (1787141347) [ 7280.930637] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7283.775275] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 08:09:12 (1787141352) [ 7292.461632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7293.232279] LustreError: 161583:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7293.366203] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7298.489179] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 7298.528264] LustreError: 161584:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7303.656398] LustreError: 161583:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7307.144268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7312.551163] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 7321.692455] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7328.564785] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7336.621212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7336.974225] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7336.983987] Lustre: Skipped 1 previous similar message [ 7344.016357] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 7352.901288] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7362.530757] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7362.533636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7362.559401] Lustre: Skipped 2 previous similar messages [ 7365.451958] Lustre: server umount lustre-MDT0000 complete [ 7367.649510] LustreError: 161577:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7367.664988] LustreError: 161577:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7369.488898] LustreError: 161564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787141440 with bad export cookie 17165256071151299454 [ 7369.496121] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7369.498992] LustreError: 161564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7372.776950] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7372.794931] Lustre: Skipped 1 previous similar message [ 7376.020420] Lustre: server umount lustre-MDT0001 complete [ 7383.737786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7397.184290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7397.425195] LustreError: 165427:0:(llog.c:1655:llog_backup()) MGC192.168.204.160@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7397.433594] Lustre: 165427:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.160@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7399.396243] LustreError: 165432:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7399.416555] LustreError: 165432:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 7416.034731] LustreError: 165433:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7416.055581] LustreError: 165433:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 7450.592312] LustreError: 165433:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7450.637097] LustreError: 165433:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 26 previous similar messages [ 7458.273912] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7458.277216] Lustre: 165427:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.204.160@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7499.269129] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7503.765225] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 7504.364939] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7513.153683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7514.336483] LustreError: 166171:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 7514.342367] LustreError: 166171:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@000000003a481e2e x1873953173926912/t0(0) o250->MGC192.168.204.160@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1787141585 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7514.395487] LustreError: 166171:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 7514.593567] LustreError: 148634:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000da56c5db x1873953173931776/t0(0) o250->MGC192.168.204.160@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7514.616276] LustreError: 163417:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7514.632767] LustreError: 163417:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 42 previous similar messages [ 7519.334159] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 7520.249915] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7527.779667] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7529.766038] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7537.027651] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7539.099972] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7561.185015] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7561.190496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7561.206944] Lustre: Skipped 2 previous similar messages [ 7561.231914] Lustre: Skipped 8 previous similar messages [ 7561.581473] Lustre: server umount lustre-MDT0000 complete [ 7565.166891] LustreError: 162318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787141635 with bad export cookie 17165256071151305908 [ 7565.173957] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7565.184837] LustreError: 162318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7565.516723] Lustre: server umount lustre-MDT0001 complete [ 7579.971091] Lustre: server umount lustre-OST0000 complete [ 7604.052897] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [ 7615.112884] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 7664.071918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7664.431783] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7664.481170] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7664.575617] Lustre: lustre-MDT0000: new disk, initializing [ 7664.670044] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7664.677682] Lustre: Skipped 1 previous similar message [ 7664.690526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7670.073155] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 7682.113649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7682.351535] Lustre: 170535:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7682.768336] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7687.980641] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 7699.111981] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7705.603288] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7713.308867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7713.580475] Lustre: lustre-OST0000: new disk, initializing [ 7713.582943] Lustre: Skipped 1 previous similar message [ 7713.585950] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7713.592093] Lustre: Skipped 2 previous similar messages [ 7713.597781] Lustre: 172503:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7720.858211] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 7721.581696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7721.601563] Lustre: Skipped 1 previous similar message [ 7721.624235] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7721.740531] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7731.601447] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7737.300637] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7737.517614] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7742.782279] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7743.051447] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7747.552435] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7747.557665] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7747.578227] LustreError: Skipped 1 previous similar message [ 7747.582430] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7747.614275] Lustre: Skipped 1 previous similar message [ 7751.973949] Lustre: server umount lustre-OST0000 complete [ 7762.931706] LustreError: 173222:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7762.945177] LustreError: 173222:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7763.196666] Lustre: server umount lustre-MDT0000 complete [ 7768.530814] LustreError: 170525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787141839 with bad export cookie 17165256071151322736 [ 7768.545817] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7768.552160] LustreError: 170525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7769.118557] Lustre: server umount lustre-MDT0001 complete [ 7780.655890] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 08:17:29 (1787141849) [ 7788.647144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7793.863596] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 7804.962365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7810.831864] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 7819.429479] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7826.311847] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7834.856395] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7842.714483] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 7852.056232] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7874.165474] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 08:19:02 (1787141942) [ 7879.655902] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7879.659790] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7879.688714] Lustre: Skipped 2 previous similar messages [ 7879.705539] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7879.717467] Lustre: Skipped 4 previous similar messages [ 7884.648181] Lustre: server umount lustre-OST0000 complete [ 7896.242460] Lustre: server umount lustre-MDT0000 complete [ 7900.498281] LustreError: 176761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787141971 with bad export cookie 17165256071151323625 [ 7900.502774] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7900.507648] LustreError: 176761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7900.918214] Lustre: server umount lustre-MDT0001 complete [ 7911.487932] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [ 7913.909637] Key type lgssc unregistered [ 7914.261701] LNet: 178675:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7914.269471] LNetError: 178675:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7914.283543] LNet: Removed LNI 192.168.204.160@tcp [ 7915.408250] Key type .llcrypt unregistered [ 7915.410781] Key type ._llcrypt unregistered [ 7935.840526] Key type ._llcrypt registered [ 7935.843472] Key type .llcrypt registered [ 7935.958541] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 7937.211417] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7937.284385] alg: No test for adler32 (adler32-zlib) [ 7938.405996] Lustre: Lustre: Build Version: 2.17.57_1_g9da61ce [ 7938.708507] LNet: Added LNI 192.168.204.160@tcp [8/256/0/180] [ 7940.356642] Key type lgssc registered [ 7941.815524] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7950.349409] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7950.364980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7965.984778] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7971.103796] LustreError: 179917:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.160@tcp: failed processing log, type 4: rc = -110 [ 7976.223572] LustreError: 179917:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7976.235047] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7976.239072] LustreError: MGC192.168.204.160@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7976.246715] LustreError: 179917:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7976.252942] LustreError: 179917:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7976.257100] LustreError: 179917:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7976.307917] Lustre: server umount lustre-OST0000 complete [ 7976.311068] LustreError: 179917:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7984.823836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7986.259416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7990.730895] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 7995.363511] LustreError: 180415:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7995.385818] LustreError: 180415:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8000.493704] LustreError: 180416:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8000.674490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8001.091527] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8006.220848] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8014.433608] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8020.754882] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8028.554634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8028.812276] LustreError: 182232:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8028.820932] Lustre: 182232:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.204.160@tcp: invalid local config log lustre-OST0000: rc = -22 [ 8028.968161] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8035.542024] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8044.835927] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8051.685774] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8051.716220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8051.716597] Lustre: Skipped 1 previous similar message [ 8051.741219] Lustre: Skipped 2 previous similar messages [ 8054.031242] Lustre: server umount lustre-MDT0000 complete [ 8056.801371] LustreError: 180416:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8056.831250] LustreError: 180416:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8057.971370] LustreError: 180398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787142128 with bad export cookie 12871230294484967027 [ 8057.973197] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8058.001098] LustreError: 180398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8058.405199] Lustre: server umount lustre-MDT0001 complete [ 8073.515952] Lustre: server umount lustre-OST0000 complete [ 8084.588682] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [ 8097.119983] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 8136.671183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8136.941411] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8136.962385] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8137.026176] Lustre: lustre-MDT0000: new disk, initializing [ 8137.096060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8137.109227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8141.612669] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8153.036772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8153.190604] Lustre: 185757:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8153.230663] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8153.233881] Lustre: Skipped 1 previous similar message [ 8153.282979] Lustre: lustre-MDT0001: new disk, initializing [ 8153.354237] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8153.395520] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8153.408869] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8157.675665] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8167.413199] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8173.749827] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8182.813231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8183.272557] Lustre: lustre-OST0000: new disk, initializing [ 8183.276698] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8183.281919] Lustre: 187727:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8183.369570] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8189.044807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8189.068673] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8189.206350] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8191.666673] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8202.448266] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8207.330058] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8207.554599] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8213.042880] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8213.325974] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8220.128282] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8220.144176] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8220.147416] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8220.163471] Lustre: Skipped 2 previous similar messages [ 8220.192297] Lustre: Skipped 1 previous similar message [ 8221.503321] Lustre: server umount lustre-OST0000 complete [ 8230.368811] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8230.372024] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8230.376267] LustreError: Skipped 1 previous similar message [ 8230.388240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8230.394678] Lustre: Skipped 1 previous similar message [ 8230.419578] Lustre: Skipped 1 previous similar message [ 8235.490455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8235.505100] Lustre: Skipped 1 previous similar message [ 8240.095140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8240.400626] Lustre: server umount lustre-MDT0000 complete [ 8240.613422] LustreError: 185768:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8240.638037] LustreError: 185768:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8244.837740] LustreError: 185747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787142315 with bad export cookie 12871230294484967748 [ 8244.848624] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8244.857956] LustreError: 185747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8245.373191] Lustre: server umount lustre-MDT0001 complete [ 8254.706657] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 08:25:23 (1787142323) [ 8273.171558] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [ 8275.723655] Key type lgssc unregistered [ 8275.979648] LNet: 190678:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8275.988658] LNetError: 190678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8277.035124] LNet: Removed LNI 192.168.204.160@tcp [ 8277.798949] Key type .llcrypt unregistered [ 8277.804075] Key type ._llcrypt unregistered [ 8296.087226] Key type ._llcrypt registered [ 8296.089695] Key type .llcrypt registered [ 8296.246102] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 8297.807070] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8297.871340] alg: No test for adler32 (adler32-zlib) [ 8299.154478] Lustre: Lustre: Build Version: 2.17.57_1_g9da61ce [ 8299.650649] LNet: Added LNI 192.168.204.160@tcp [8/256/0/180] [ 8301.359569] Key type lgssc registered [ 8302.284975] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8313.032870] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8313.049193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8314.462658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8319.403724] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8324.580124] LustreError: 191897:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8324.603789] LustreError: 191897:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8328.615820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8329.044897] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8333.637365] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8341.536373] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8348.293753] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8356.880520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8357.077136] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8357.079664] LustreError: 193762:0:(llog.c:1696:llog_backup()) MGC192.168.204.160@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8357.085330] Lustre: 193762:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.204.160@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8357.215534] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8364.087676] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8374.285969] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8378.858889] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8378.868440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8378.869452] Lustre: Skipped 2 previous similar messages [ 8378.896360] Lustre: Skipped 2 previous similar messages [ 8383.676289] Lustre: server umount lustre-MDT0000 complete [ 8383.982403] LustreError: 191893:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8384.006333] LustreError: 191893:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8387.861789] LustreError: 192633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787142458 with bad export cookie 14759685412790996695 [ 8387.870946] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8387.872300] LustreError: 192633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8388.280900] Lustre: server umount lustre-MDT0001 complete [ 8403.069303] Lustre: server umount lustre-OST0000 complete [ 8415.807615] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [ 8429.380313] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 8469.074403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8469.280951] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8469.299128] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8469.350865] Lustre: lustre-MDT0000: new disk, initializing [ 8469.413038] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8469.432023] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8474.240839] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8485.378145] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8485.488769] Lustre: 197284:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8485.521357] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8485.523548] Lustre: Skipped 1 previous similar message [ 8485.525737] LustreError: 198069:0:(llog.c:1696:llog_backup()) MGC192.168.204.160@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8485.532506] LustreError: 198069:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8485.536643] Lustre: 198069:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.160@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8485.557830] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8485.563505] Lustre: Skipped 1 previous similar message [ 8485.631471] Lustre: lustre-MDT0001: new disk, initializing [ 8485.706813] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8485.742387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8485.756595] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8490.761890] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8500.719958] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8507.050101] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8513.930032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8514.308930] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8514.311078] Lustre: Skipped 1 previous similar message [ 8514.314905] LustreError: 199232:0:(llog.c:1696:llog_backup()) MGC192.168.204.160@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8514.329333] LustreError: 199232:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8514.335953] Lustre: 199232:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.160@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8514.347415] Lustre: 199232:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8514.397186] Lustre: lustre-OST0000: new disk, initializing [ 8514.402486] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8514.409617] Lustre: 199257:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8514.553191] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8520.809491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8520.814555] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8520.964331] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8521.631622] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8532.936632] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8538.422813] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8538.641236] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8544.290315] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8544.517534] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8546.787071] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8546.792118] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8546.800362] LustreError: Skipped 1 previous similar message [ 8546.804454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8546.818061] Lustre: Skipped 1 previous similar message [ 8546.847478] Lustre: Skipped 1 previous similar message [ 8551.911875] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8551.920364] Lustre: Skipped 1 previous similar message [ 8553.001892] Lustre: server umount lustre-OST0000 complete [ 8557.544160] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8557.553682] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8557.575058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8562.145778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8562.152120] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8563.911449] Lustre: server umount lustre-MDT0000 complete [ 8567.265076] LustreError: 198549:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8567.290790] LustreError: 198549:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8567.878709] LustreError: 198082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787142638 with bad export cookie 14759685412790997416 [ 8567.893652] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8567.897336] LustreError: 198082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8568.245241] Lustre: server umount lustre-MDT0001 complete [ 8579.151897] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 08:30:47 (1787142647) [ 8596.287955] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 8606.418697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8606.785613] LustreError: 202387:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8606.824866] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8611.233858] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8612.325195] LustreError: 202388:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8621.510887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8621.540433] LustreError: 202388:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8621.558627] LustreError: 202388:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8626.178053] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8629.645596] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8637.122593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8644.629702] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8655.428083] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8655.765726] Lustre: lustre-OST0001: new disk, initializing [ 8655.776957] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8655.787827] Lustre: 204696:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8655.960901] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8655.972304] Lustre: Skipped 2 previous similar messages [ 8663.060900] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8664.719806] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8664.737648] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8664.894966] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8673.619615] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8684.058735] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8686.371449] Lustre: DEBUG MARKER: ADD OST3 [ 8691.407349] loop: module loaded [ 8691.516726] loop0: detected capacity change from 0 to 204800000 [ 8691.740313] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8691.774393] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8691.791162] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8698.757214] loop0: detected capacity change from 0 to 204800000 [ 8699.273267] Lustre: DEBUG MARKER: STOP OST3 [ 8705.904130] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8706.285595] Lustre: lustre-OST0003: new disk, initializing [ 8706.293988] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8706.300930] Lustre: 206731:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8707.481880] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8707.593065] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8707.605311] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8707.648412] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8709.546077] LustreError: 206827:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8710.366693] Lustre: Failing over lustre-OST0003 [ 8710.620459] Lustre: server umount lustre-OST0003 complete [ 8712.679895] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8712.706730] Lustre: Skipped 1 previous similar message [ 8712.712797] LustreError: 203873:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8712.725729] LustreError: 203873:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8714.450219] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8723.303651] Lustre: DEBUG MARKER: START OST3 again [ 8729.946085] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8730.314253] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8730.322506] Lustre: Skipped 1 previous similar message [ 8730.343769] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8731.691495] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8731.892053] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8731.892540] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8736.334560] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8757.569279] Lustre: Failing over lustre-OST0003 [ 8757.653954] Lustre: server umount lustre-OST0003 complete [ 8757.733686] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8757.740466] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8757.744520] LustreError: Skipped 1 previous similar message [ 8757.757392] LustreError: 205130:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8757.787418] LustreError: 205130:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 8775.649687] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8775.658513] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8775.675674] Lustre: Skipped 1 previous similar message [ 8775.679772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8775.685765] Lustre: Skipped 1 previous similar message [ 8781.203729] Lustre: server umount lustre-MDT0000 complete [ 8784.460748] LustreError: 203896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787142855 with bad export cookie 14759685412790998298 [ 8784.476849] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8790.899929] Lustre: server umount lustre-MDT0001 complete [ 8795.163050] Lustre: server umount lustre-OST0000 complete [ 8799.113720] Lustre: server umount lustre-OST0001 complete [ 8805.348721] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [ 8813.877655] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 8852.243022] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8852.439645] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8852.470652] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8852.561534] Lustre: lustre-MDT0000: new disk, initializing [ 8852.717991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8857.300941] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8868.551141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8868.639924] Lustre: 212152:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8868.684918] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8868.687353] Lustre: Skipped 1 previous similar message [ 8868.740919] Lustre: lustre-MDT0001: new disk, initializing [ 8868.815877] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8868.822731] Lustre: Skipped 1 previous similar message [ 8868.847880] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8868.862227] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8873.641234] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8884.777797] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8891.216060] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8899.604192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8899.899173] Lustre: 214122:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8901.769470] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8901.778421] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8901.876040] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8906.204885] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8915.606963] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8920.635801] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8920.849255] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8925.198780] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8925.437201] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8932.845296] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8932.847539] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8932.854931] LustreError: Skipped 1 previous similar message [ 8932.869718] Lustre: Skipped 6 previous similar messages [ 8932.873654] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8932.878123] Lustre: Skipped 8 previous similar messages [ 8934.320946] Lustre: server umount lustre-OST0000 complete [ 8944.843434] Lustre: server umount lustre-MDT0000 complete [ 8948.199276] LustreError: 213432:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8948.223381] LustreError: 213432:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 17 previous similar messages [ 8949.474233] LustreError: 212145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787143020 with bad export cookie 14759685412791000587 [ 8949.477145] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8949.486674] LustreError: 212145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8949.976567] Lustre: server umount lustre-MDT0001 complete [ 8953.981695] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 08:37:02 (1787143022) [ 8972.170149] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [ 8981.294768] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 9015.903826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9016.195217] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9016.254279] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9016.259079] Lustre: Skipped 1 previous similar message [ 9016.328972] Lustre: lustre-MDT0000: new disk, initializing [ 9016.333060] Lustre: Skipped 1 previous similar message [ 9016.429553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9021.379421] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9032.397830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9032.548067] Lustre: 218733:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9032.949089] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9037.819425] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9048.506496] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9053.983211] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9060.974456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9061.304521] Lustre: 220701:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9063.440785] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9068.092976] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9077.947612] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9082.483517] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9082.766213] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9087.158752] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9087.366799] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9094.123305] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9094.136262] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9094.146707] Lustre: Skipped 2 previous similar messages [ 9094.155336] LustreError: Skipped 1 previous similar message [ 9094.162917] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9094.176683] Lustre: Skipped 5 previous similar messages [ 9095.586856] Lustre: server umount lustre-OST0000 complete [ 9106.382341] Lustre: server umount lustre-MDT0000 complete [ 9109.473227] LustreError: 218744:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9109.501233] LustreError: 218744:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9110.403548] LustreError: 218725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787143181 with bad export cookie 14759685412791001476 [ 9110.411533] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9110.419383] LustreError: 218725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9110.990490] Lustre: server umount lustre-MDT0001 complete [ 9120.507492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9125.678967] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9134.133652] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9134.484912] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9134.494330] Lustre: Skipped 5 previous similar messages [ 9138.541157] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9145.634454] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9151.326261] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9158.289887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9164.329719] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9173.147682] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9185.762047] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9185.771548] LustreError: Skipped 1 previous similar message [ 9185.775649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9185.778675] Lustre: Skipped 2 previous similar messages [ 9189.738097] Lustre: server umount lustre-OST0000 complete [ 9199.413724] Lustre: server umount lustre-MDT0000 complete [ 9203.369177] LustreError: 222901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787143274 with bad export cookie 14759685412791002351 [ 9203.376098] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9203.386432] LustreError: 222901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 9203.721492] Lustre: server umount lustre-MDT0001 complete [ 9311.170976] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [ 9314.256765] Key type lgssc unregistered [ 9314.575172] LNet: 226587:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9314.587698] LNetError: 226587:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9314.603663] LNet: Removed LNI 192.168.204.160@tcp [ 9315.589627] Key type .llcrypt unregistered [ 9315.592597] Key type ._llcrypt unregistered [ 9343.509515] Key type ._llcrypt registered [ 9343.512962] Key type .llcrypt registered [ 9343.631596] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [ 9360.945422] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 9361.654881] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9361.775342] alg: No test for adler32 (adler32-zlib) [ 9362.986555] Lustre: Lustre: Build Version: 2.17.57_1_g9da61ce [ 9363.179438] LNet: Added LNI 192.168.204.160@tcp [8/256/0/180] [ 9364.855398] Key type lgssc registered [ 9366.283126] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9403.054872] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9403.078818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9404.307931] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9404.336590] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9404.398823] Lustre: lustre-MDT0000: new disk, initializing [ 9404.462971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9404.489810] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9408.699117] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9419.162562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9419.292626] Lustre: 229939:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9419.367037] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9419.375733] Lustre: Skipped 1 previous similar message [ 9419.501676] Lustre: lustre-MDT0001: new disk, initializing [ 9419.625722] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9419.678594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9419.693444] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9424.062567] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9433.000409] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9437.888331] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9444.453152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9444.760469] Lustre: lustre-OST0000: new disk, initializing [ 9444.764309] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9444.777844] Lustre: 231907:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9444.881406] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9450.693944] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9453.114265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9453.128547] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9453.258174] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9460.343393] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9465.474914] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9465.684902] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9470.531515] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9470.710586] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9474.016791] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9474.023893] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9474.031260] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9474.050710] Lustre: Skipped 1 previous similar message [ 9478.896892] Lustre: server umount lustre-OST0000 complete [ 9484.259663] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9484.276233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9484.278853] Lustre: Skipped 1 previous similar message [ 9488.969353] Lustre: server umount lustre-MDT0000 complete [ 9492.510826] LustreError: 229931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787143563 with bad export cookie 14270004755344983767 [ 9492.516797] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9492.529541] LustreError: 229931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9492.850510] Lustre: server umount lustre-MDT0001 complete [ 9511.052799] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 9521.489492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9521.814325] LustreError: 234846:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9521.834556] LustreError: 234846:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9521.855989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9525.799796] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9527.267869] LustreError: 234847:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9531.369514] LustreError: 234846:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9533.825780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9534.219441] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9538.671696] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9541.605916] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9548.182224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9548.513456] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9553.644255] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9561.402548] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9561.575346] Lustre: lustre-OST0001: new disk, initializing [ 9561.578797] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9561.584745] Lustre: 237155:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9563.755453] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9563.770164] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9563.873146] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9568.625379] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9580.715715] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9588.523501] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9601.862766] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 08:47:50 (1787143670) [ 9606.117055] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9606.130171] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9606.143711] Lustre: Skipped 1 previous similar message [ 9606.153079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9608.161488] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9608.165199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9608.180795] Lustre: Skipped 1 previous similar message [ 9608.199779] Lustre: Skipped 3 previous similar messages [ 9610.486613] Lustre: server umount lustre-MDT0000 complete [ 9613.289699] LustreError: 234846:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9613.313925] LustreError: 234846:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9614.084904] LustreError: 237154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787143684 with bad export cookie 14270004755344984649 [ 9614.098472] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9614.102992] LustreError: 237154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9614.541594] Lustre: server umount lustre-MDT0001 complete [ 9628.984897] Lustre: server umount lustre-OST0000 complete [ 9634.280099] Lustre: 228163:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787143689/real 1787143689] req@ffff9ebcc7cee300 x1873955712513280/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787143705 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9634.319833] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9634.348683] Lustre: Skipped 1 previous similar message [ 9634.681932] Lustre: server umount lustre-OST0001 complete [ 9643.794723] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [ 9657.189274] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 9698.285461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9698.661591] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9698.721230] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9698.886879] Lustre: lustre-MDT0000: new disk, initializing [ 9699.024635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9699.040035] Lustre: Skipped 1 previous similar message [ 9699.061481] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9703.894255] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9713.941105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9714.043473] Lustre: 241101:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9714.084507] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9714.087574] Lustre: Skipped 1 previous similar message [ 9714.170050] Lustre: lustre-MDT0001: new disk, initializing [ 9714.352207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9714.364279] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9718.121864] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9727.607945] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9733.290866] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9740.352709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9740.595552] Lustre: lustre-OST0000: new disk, initializing [ 9740.598915] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9740.604355] Lustre: 243069:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9740.682466] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9740.692824] Lustre: Skipped 1 previous similar message [ 9742.259611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9742.282742] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9742.399462] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9747.105876] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9756.227508] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9760.279097] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9760.489130] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9764.849930] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9765.079914] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9767.909478] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9767.922251] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9767.950813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9767.957443] Lustre: Skipped 1 previous similar message [ 9778.148365] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9778.163736] Lustre: Skipped 3 previous similar messages [ 9781.215131] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9781.390892] Lustre: server umount lustre-OST0000 complete [ 9785.826875] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9785.833886] LustreError: Skipped 1 previous similar message [ 9785.843340] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9785.860978] Lustre: Skipped 1 previous similar message [ 9791.807540] Lustre: server umount lustre-MDT0000 complete [ 9793.506205] LustreError: 241109:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9793.529269] LustreError: 241109:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9796.163573] LustreError: 241092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787143866 with bad export cookie 14270004755344986784 [ 9796.179169] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9796.188030] LustreError: 241092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9796.770451] Lustre: server umount lustre-MDT0001 complete [ 9806.280721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9806.667750] LustreError: 245287:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9806.729580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9811.638860] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9821.686817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9827.881682] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9836.568594] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9843.343833] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9849.974609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9857.252716] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9866.261385] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10015.719225] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10015.720402] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10015.732038] LustreError: Skipped 1 previous similar message [10015.740700] Lustre: Skipped 1 previous similar message [10015.741929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10015.754710] Lustre: Skipped 4 previous similar messages [10017.120679] Lustre: server umount lustre-OST0000 complete [10035.167305] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10035.391155] Lustre: server umount lustre-MDT0000 complete [10036.212904] LustreError: 246018:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10036.252285] LustreError: 246018:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10039.522498] LustreError: 245268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787144110 with bad export cookie 14270004755344987666 [10039.531296] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10039.858968] Lustre: server umount lustre-MDT0001 complete [10135.553442] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [10138.485958] Key type lgssc unregistered [10138.769666] LNet: 248901:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10138.778360] LNetError: 248901:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10138.804403] LNet: Removed LNI 192.168.204.160@tcp [10139.851197] Key type .llcrypt unregistered [10139.858239] Key type ._llcrypt unregistered [10166.725203] Key type ._llcrypt registered [10166.727344] Key type .llcrypt registered [10166.836922] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [10181.956573] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [10182.837354] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10182.913394] alg: No test for adler32 (adler32-zlib) [10183.966589] Lustre: Lustre: Build Version: 2.17.57_1_g9da61ce [10184.172110] LNet: Added LNI 192.168.204.160@tcp [8/256/0/180] [10185.855207] Key type lgssc registered [10186.882262] Lustre: Echo OBD driver; http://www.lustre.org/ [10237.854739] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [10251.633072] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10251.706822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10253.030579] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10253.141813] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10253.246349] Lustre: lustre-MDT0000: new disk, initializing [10253.362587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10253.380898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10258.359075] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [10271.107316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10271.258540] Lustre: 252975:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10271.343165] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10271.349180] Lustre: Skipped 1 previous similar message [10271.436507] Lustre: lustre-MDT0001: new disk, initializing [10271.560844] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10271.598674] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10271.616224] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10276.820563] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [10282.011595] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10289.628976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10289.974553] Lustre: lustre-OST0000: new disk, initializing [10290.005733] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10290.013940] Lustre: 254608:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10290.120621] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10292.584133] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10292.593631] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10292.767848] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10297.742536] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [10309.663402] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10309.853502] Lustre: lustre-OST0001: new disk, initializing [10309.863554] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10309.874351] Lustre: 255479:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10310.010606] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10316.362977] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [10317.940808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10317.955411] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10318.040921] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10328.120556] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10335.620149] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10353.640929] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10353.660272] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10353.675631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10354.163773] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10354.181120] Lustre: Skipped 2 previous similar messages [10354.191826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10354.203924] Lustre: Skipped 3 previous similar messages [10355.788841] Lustre: server umount lustre-MDT0000 complete [10359.314714] LustreError: 253771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787144430 with bad export cookie 13945927722791712157 [10359.325511] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10359.332429] LustreError: 253771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10359.678978] Lustre: server umount lustre-MDT0001 complete [10373.784764] Lustre: server umount lustre-OST0000 complete [10388.975943] Lustre: server umount lustre-OST0001 complete [10404.675363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10415.122402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10424.414526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10435.411830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10455.361581] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [10466.222758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10466.263306] Lustre: lustre-MDT0000: reset Object Index mappings [10466.745644] LustreError: 260500:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10466.760310] LustreError: 260500:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10466.860054] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10471.905109] LustreError: 260501:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10473.179536] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [10477.041922] LustreError: 260500:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10482.156076] LustreError: 260501:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10483.726756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10483.758525] Lustre: lustre-MDT0001: reset Object Index mappings [10484.302735] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10489.468707] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [10493.017382] Lustre: 261644:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10502.593610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10503.187746] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10508.337080] LustreError: 261999:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10511.432026] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10512.479727] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [10516.454993] LustreError: 261999:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10516.484484] LustreError: 261999:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10524.798707] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10530.819708] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10534.001455] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [10543.624634] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10548.435121] Lustre: 263517:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10555.877959] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10555.889198] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10555.912828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10556.061934] Lustre: server umount lustre-MDT0000 complete [10556.398527] LustreError: 261233:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10556.409762] LustreError: 261233:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10560.998396] LustreError: 261245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787144631 with bad export cookie 13945927722791718947 [10561.000286] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10561.016202] LustreError: 261245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10561.620509] Lustre: server umount lustre-MDT0001 complete [10577.823216] Lustre: 250480:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787144632/real 1787144632] req@ffff9ebcc7cebb80 x1873956573415552/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787144648 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10577.844722] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10577.853163] Lustre: Skipped 3 previous similar messages [10577.971578] Lustre: server umount lustre-OST0000 complete [10581.921099] Lustre: 250479:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787144636/real 1787144636] req@ffff9ebbc3bf6a00 x1873956573415808/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787144652 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10582.987278] Lustre: server umount lustre-OST0001 complete [10603.008625] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 09:04:31 (1787144671) [10623.477408] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [10635.588190] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [10676.334786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10676.569788] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10676.593727] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10676.662622] Lustre: lustre-MDT0000: new disk, initializing [10676.759265] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10676.775454] Lustre: Skipped 1 previous similar message [10676.793579] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10682.417291] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [10693.227442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10693.370275] Lustre: 267771:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10693.422317] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10693.426169] Lustre: Skipped 1 previous similar message [10693.482320] Lustre: lustre-MDT0001: new disk, initializing [10693.632328] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10693.643157] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10697.998445] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [10708.076562] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10714.051147] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10722.564069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10722.944121] Lustre: lustre-OST0000: new disk, initializing [10722.949955] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10722.958969] Lustre: 269739:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10724.234229] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10724.248984] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10724.419112] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10731.527636] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [10744.297377] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10751.290627] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10751.597959] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10757.107272] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10757.434156] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10760.160069] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10760.169413] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10760.177823] LustreError: Skipped 1 previous similar message [10760.205432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10765.298762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10765.307740] Lustre: Skipped 1 previous similar message [10765.904785] Lustre: server umount lustre-OST0000 complete [10770.400220] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10770.411493] Lustre: Skipped 2 previous similar messages [10775.522903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10775.531566] Lustre: Skipped 4 previous similar messages [10776.067885] Lustre: server umount lustre-MDT0000 complete [10780.644237] LustreError: 267784:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10780.668746] LustreError: 267784:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [10781.462420] LustreError: 267765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787144852 with bad export cookie 13945927722791721040 [10781.495091] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10782.101620] Lustre: server umount lustre-MDT0001 complete [10802.763957] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [10813.748669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10814.312083] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10814.324358] Lustre: Skipped 2 previous similar messages [10820.224625] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [10829.706262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10835.607075] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [10839.477152] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10846.797572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10853.298212] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [10861.825185] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10862.023641] Lustre: lustre-OST0001: new disk, initializing [10862.027578] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10862.034774] Lustre: 275003:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10864.112805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10864.118553] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10864.250528] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10870.360813] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [10880.893334] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10885.178356] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10892.103121] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10892.187640] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10901.985111] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10902.004119] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10902.016827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10909.352043] Lustre: server umount lustre-MDT0000 complete [10910.186158] LustreError: 272695:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10910.202108] LustreError: 272695:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [10914.365531] LustreError: 272676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787144985 with bad export cookie 13945927722791721922 [10914.369073] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10914.380141] LustreError: 272676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [10914.716374] Lustre: server umount lustre-MDT0001 complete [10929.283175] Lustre: server umount lustre-OST0000 complete [10932.255111] Lustre: 250481:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787144986/real 1787144986] req@ffff9ebcc48f2300 x1873956573529984/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787145002 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10933.435835] Lustre: server umount lustre-OST0001 complete [10954.649908] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [10968.405956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10968.994618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10968.999931] Lustre: Skipped 3 previous similar messages [10974.089985] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [10985.195985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10991.532340] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [10994.759757] Lustre: 279207:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11002.270335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11010.548788] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [11014.274049] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [11019.573459] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11025.420874] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [11026.726807] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [11034.735821] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11039.441344] Lustre: 281083:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11045.435627] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [11045.478839] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [11057.025261] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 09:12:05 (1787145125) [11076.590718] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11076.606954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11076.617565] Lustre: Skipped 6 previous similar messages [11076.643912] Lustre: Skipped 7 previous similar messages [11079.959530] Lustre: server umount lustre-MDT0000 complete [11081.709075] LustreError: 278066:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11081.724554] LustreError: 278066:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [11084.953594] LustreError: 281086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787145155 with bad export cookie 13945927722791728509 [11084.955533] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11084.967206] LustreError: 281086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11091.496941] Lustre: server umount lustre-MDT0001 complete [11110.879498] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11111.221024] Lustre: server umount lustre-OST0000 complete [11115.886959] Lustre: server umount lustre-OST0001 complete [11124.490181] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [11137.102483] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [11188.378653] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [11198.922721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11199.184724] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11199.225426] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11199.306274] Lustre: lustre-MDT0000: new disk, initializing [11199.430147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11204.738536] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [11216.524056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11216.691832] Lustre: 285327:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11217.087838] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11222.146471] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [11235.387444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11235.849416] Lustre: 286960:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11235.971310] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11235.974870] Lustre: Skipped 5 previous similar messages [11237.468137] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11242.723293] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [11254.138107] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11254.396845] Lustre: 287832:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11255.578513] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11255.592663] Lustre: Skipped 1 previous similar message [11255.700123] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11261.347367] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [11271.923493] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11276.858697] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11276.867675] Lustre: Skipped 1 previous similar message [11293.847338] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 09:16:02 (1787145362) [11296.737750] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11296.743146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11296.758354] Lustre: Skipped 6 previous similar messages [11296.770818] Lustre: Skipped 6 previous similar messages [11298.728209] Lustre: server umount lustre-MDT0000 complete [11302.223946] LustreError: 286959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787145372 with bad export cookie 13945927722791736139 [11302.230239] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11302.236440] LustreError: 286959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11302.639580] Lustre: server umount lustre-MDT0001 complete [11317.356104] Lustre: server umount lustre-OST0000 complete [11332.228288] Lustre: server umount lustre-OST0001 complete [11343.447518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11344.037667] LustreError: 290081:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11344.056404] LustreError: 290081:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [11349.364266] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [11359.783614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11365.765786] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [11374.837272] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11383.143981] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11391.391312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11400.515774] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [11409.375105] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11432.529433] loop1: detected capacity change from 0 to 204800000 [11432.765154] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11432.779433] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11432.792433] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11438.464401] loop1: detected capacity change from 0 to 204800000 [11445.312034] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11445.482516] LustreError: 290070:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs25572: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11445.497585] LustreError: 290070:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs25572 [11445.506609] LustreError: 290070:0:(mgs_handler.c:517:mgs_target_reg()) fs25572-MDT0001: Error checking target index: rc = -13 [11445.533358] LustreError: fs25572-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11445.556557] LustreError: 293804:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11445.574132] LustreError: 293804:0:(tgt_mount.c:2129:server_put_super()) no obd fs25572-MDT0001 [11445.581368] LustreError: 293804:0:(tgt_mount.c:116:server_deregister_mount()) fs25572-MDT0001 not registered [11445.671459] Lustre: server umount fs25572-MDT0001 complete [11445.679444] LustreError: 293804:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11455.287578] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11455.381185] Lustre: Setting parameter fs25572-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs25572-MDT0001 [11455.441803] Lustre: srv-fs25572-MDT0001: No data found on store. Initialize space. [11455.444598] Lustre: Skipped 4 previous similar messages [11455.547891] Lustre: fs25572-MDT0001: new disk, initializing [11455.555296] Lustre: Skipped 3 previous similar messages [11455.714882] Lustre: srv-fs25572-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11455.723190] Lustre: 294271:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs25572-MDT0001-mdtlov: can't create new llog: rc = -115 [11455.738584] LustreError: 294271:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs25572-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11460.770398] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [11468.436126] Lustre: server umount fs25572-MDT0001 complete [11482.952262] loop1: detected capacity change from 0 to 204800000 [11485.185369] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11489.521055] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11490.079482] Lustre: 295840:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs25572-MDT0001-mdtlov: can't create new llog: rc = -115 [11490.090316] LustreError: 295840:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs25572-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11495.052806] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [11500.083656] Lustre: server umount fs25572-MDT0001 complete [11510.756288] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11510.760809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11510.768033] Lustre: Skipped 1 previous similar message [11510.782405] Lustre: Skipped 3 previous similar messages [11515.765939] Lustre: server umount lustre-MDT0000 complete [11524.454366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11524.619776] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11529.482482] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [11530.221238] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11537.275593] loop1: detected capacity change from 0 to 204800000 [11542.441700] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11542.840339] Lustre: 298286:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs25572-MDT0001-mdtlov: can't create new llog: rc = -115 [11542.848326] LustreError: 298286:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs25572-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11547.423825] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [11552.692880] Lustre: server umount fs25572-MDT0001 complete [11576.299984] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11576.314403] LustreError: Skipped 1 previous similar message [11580.831789] Lustre: server umount lustre-OST0000 complete [11599.327335] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11599.609169] Lustre: server umount lustre-MDT0000 complete [11603.273023] LustreError: 290061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787145674 with bad export cookie 13945927722791738820 [11603.283977] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11603.288846] LustreError: 290061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11603.666511] Lustre: server umount lustre-MDT0001 complete [11614.970636] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [11618.234062] Key type lgssc unregistered [11618.562693] LNet: 300634:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11618.580543] LNetError: 300634:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11618.608292] LNet: Removed LNI 192.168.204.160@tcp [11619.443300] Key type .llcrypt unregistered [11619.447061] Key type ._llcrypt unregistered [11642.658655] Key type ._llcrypt registered [11642.665648] Key type .llcrypt registered [11642.857533] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [11644.340333] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11644.463312] alg: No test for adler32 (adler32-zlib) [11645.492702] Lustre: Lustre: Build Version: 2.17.57_1_g9da61ce [11645.686444] LNet: Added LNI 192.168.204.160@tcp [8/256/0/180] [11647.455229] Key type lgssc registered [11648.401735] Lustre: Echo OBD driver; http://www.lustre.org/ [11656.053590] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11656.096526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11657.758932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11662.492338] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [11667.936864] LustreError: 301906:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11667.961966] LustreError: 301906:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11668.967738] LustreError: 301907:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11672.981747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11673.527449] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11678.644972] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [11687.991800] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11694.148654] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11701.301724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11701.798264] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11703.851497] LustreError: 303735:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11708.921824] LustreError: 303737:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11709.294337] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [11714.035137] LustreError: 303856:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11717.552786] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11724.278737] LustreError: 303738:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11724.299259] LustreError: 303738:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [11737.647726] loop1: detected capacity change from 0 to 204800000 [11737.842793] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11737.862347] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11737.891130] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11740.904721] LustreError: 303856:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11740.939989] LustreError: 303856:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [11744.136233] loop1: detected capacity change from 0 to 204800000 [11751.799501] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11752.018375] Lustre: fs2418-OST0000: new disk, initializing [11752.020913] Lustre: srv-fs2418-OST0000: No data found on store. Initialize space. [11752.035803] Lustre: 305646:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11752.152252] Lustre: fs2418-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11760.023456] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [11769.317132] Lustre: server umount fs2418-OST0000 complete [11775.466093] LustreError: 303856:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11775.487989] LustreError: 303856:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 19 previous similar messages [11783.009063] loop1: detected capacity change from 0 to 204800000 [11785.725179] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11789.104459] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11789.493516] Lustre: fs2418-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11797.284669] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [11801.076407] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11806.818982] loop2: detected capacity change from 0 to 204800000 [11807.038736] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11807.048432] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11807.072211] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11812.825198] loop2: detected capacity change from 0 to 204800000 [11818.878149] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11819.058161] LustreError: fs2418-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11819.070688] LustreError: fs2418-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11819.090957] LustreError: 308688:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11819.114156] LustreError: 308688:0:(tgt_mount.c:2129:server_put_super()) no obd fs2418-OST0001 [11819.120801] LustreError: 308688:0:(tgt_mount.c:116:server_deregister_mount()) fs2418-OST0001 not registered [11819.194983] Lustre: server umount fs2418-OST0001 complete [11819.203452] LustreError: 308688:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11832.197727] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11832.537098] Lustre: fs2418-OST0001: new disk, initializing [11832.550709] Lustre: srv-fs2418-OST0001: No data found on store. Initialize space. [11832.563906] Lustre: 309238:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11832.720700] Lustre: fs2418-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11839.728319] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [11842.025904] LustreError: 303737:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11842.051202] LustreError: 303737:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 38 previous similar messages [11847.723357] Lustre: server umount fs2418-OST0000 complete [11856.835893] Lustre: server umount fs2418-OST0001 complete [11871.152354] loop1: detected capacity change from 0 to 204800000 [11878.927664] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11879.328681] Lustre: fs2418-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11888.290839] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [11895.425317] loop2: detected capacity change from 0 to 204800000 [11901.593931] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11908.550975] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [11919.426295] Lustre: server umount fs2418-OST0000 complete [11929.870857] Lustre: server umount fs2418-OST0001 complete [11939.328235] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11949.537445] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11949.550335] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11949.558519] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11949.570031] Lustre: Skipped 1 previous similar message [11954.043835] Lustre: server umount lustre-OST0000 complete [11959.779778] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11959.813832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11959.821716] Lustre: Skipped 1 previous similar message [11964.906054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11964.915144] Lustre: Skipped 1 previous similar message [11965.274431] Lustre: server umount lustre-MDT0000 complete [11968.905297] LustreError: 301888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787146039 with bad export cookie 16124790637015094589 [11968.922303] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11968.924179] LustreError: 301888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11969.249685] Lustre: server umount lustre-MDT0001 complete [11981.478484] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [11984.187759] Key type lgssc unregistered [11984.575181] LNet: 315167:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11984.591136] LNetError: 315167:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11984.604915] LNet: Removed LNI 192.168.204.160@tcp [11985.527223] Key type .llcrypt unregistered [11985.532036] Key type ._llcrypt unregistered [12020.029627] Key type ._llcrypt registered [12020.034649] Key type .llcrypt registered [12020.193345] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [12036.216843] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [12037.399057] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12037.440614] alg: No test for adler32 (adler32-zlib) [12038.519583] Lustre: Lustre: Build Version: 2.17.57_1_g9da61ce [12038.777858] LNet: Added LNI 192.168.204.160@tcp [8/256/0/180] [12040.511200] Key type lgssc registered [12041.670326] Lustre: Echo OBD driver; http://www.lustre.org/ [12076.778217] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12076.795627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12078.033627] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12078.075623] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12078.134653] Lustre: lustre-MDT0000: new disk, initializing [12078.210740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12078.224198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12082.151781] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [12092.080265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12092.240839] Lustre: 319031:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12092.310356] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12092.320767] Lustre: Skipped 1 previous similar message [12092.448670] Lustre: lustre-MDT0001: new disk, initializing [12092.538898] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12092.584945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12092.590661] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12097.153488] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [12105.534837] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12110.604985] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12116.641258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12116.833423] Lustre: lustre-OST0000: new disk, initializing [12116.836426] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12116.841065] Lustre: 320997:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12116.933889] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12122.949172] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [12125.238221] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12125.249818] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12125.333187] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12131.773427] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12133.952659] Lustre: Mounted lustre-client - version 2.17.57_1_g9da61ce [12134.891092] Lustre: Unmounted lustre-client [12146.145579] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12146.153655] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12146.166711] LustreError: Skipped 1 previous similar message [12146.176177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12146.196779] Lustre: Skipped 1 previous similar message [12146.229629] Lustre: Skipped 1 previous similar message [12149.225100] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12149.234960] Lustre: Skipped 1 previous similar message [12150.471066] Lustre: server umount lustre-OST0000 complete [12158.944733] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12158.960382] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12158.991584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12161.012464] Lustre: server umount lustre-MDT0000 complete [12164.908525] LustreError: 319022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787146235 with bad export cookie 12914022996071149075 [12164.912874] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12164.919475] LustreError: 319022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12165.437415] Lustre: server umount lustre-MDT0001 complete [12177.680746] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [12180.615486] Key type lgssc unregistered [12194.463218] LNet: 1 local NIs in recovery (showing 1): 192.168.204.160@tcp [12195.296646] LNet: 323159:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12195.314890] LNetError: 323159:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12264.997696] LNet: Removed LNI 192.168.204.160@tcp [12265.770781] Key type .llcrypt unregistered [12265.775142] Key type ._llcrypt unregistered [12281.579326] Key type ._llcrypt registered [12281.585113] Key type .llcrypt registered [12281.704593] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [12297.359829] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [12298.099589] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12298.122314] alg: No test for adler32 (adler32-zlib) [12299.181659] Lustre: Lustre: Build Version: 2.17.57_1_g9da61ce [12299.456371] LNet: Added LNI 192.168.204.160@tcp [8/256/0/180] [12301.147511] Key type lgssc registered [12302.265689] Lustre: Echo OBD driver; http://www.lustre.org/ [12333.618859] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 09:33:22 (1787146402) [12340.107884] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12340.158064] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12341.427733] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12341.474475] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12341.573981] Lustre: lustre-MDT0000: new disk, initializing [12341.676967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12341.694909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12346.402427] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [12356.819667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12356.936610] Lustre: 326382:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12356.984115] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12356.987798] Lustre: Skipped 1 previous similar message [12357.079835] Lustre: lustre-MDT0001: new disk, initializing [12357.150335] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12357.189053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12357.198224] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12361.043972] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [12369.042132] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12374.651876] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12381.553186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12381.857682] Lustre: lustre-OST0000: new disk, initializing [12381.860595] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12381.866909] Lustre: 328351:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12381.943972] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12387.571930] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [12390.969937] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12390.981339] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12391.139416] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12397.401963] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12416.997439] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12417.003064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12417.003786] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12417.023784] Lustre: Skipped 1 previous similar message [12417.040506] Lustre: Skipped 1 previous similar message [12419.129265] Lustre: server umount lustre-OST0000 complete [12423.649038] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12423.658661] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12423.669217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12427.232811] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12427.234287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12427.262230] Lustre: Skipped 1 previous similar message [12428.955386] Lustre: server umount lustre-MDT0000 complete [12432.833474] LustreError: 327178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787146503 with bad export cookie 8848839466361136229 [12432.842047] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12432.863921] LustreError: 327178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12433.283660] Lustre: server umount lustre-MDT0001 complete [12444.067172] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [12447.400247] Key type lgssc unregistered [12447.744392] LNet: 330261:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12447.764515] LNetError: 330261:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12447.777205] LNet: Removed LNI 192.168.204.160@tcp [12448.535026] Key type .llcrypt unregistered [12448.541075] Key type ._llcrypt unregistered [12468.466343] Key type ._llcrypt registered [12468.468354] Key type .llcrypt registered [12468.601924] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [12469.847623] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12469.887449] alg: No test for adler32 (adler32-zlib) [12470.932360] Lustre: Lustre: Build Version: 2.17.57_1_g9da61ce [12471.186895] LNet: Added LNI 192.168.204.160@tcp [8/256/0/180] [12472.911139] Key type lgssc registered [12473.982606] Lustre: Echo OBD driver; http://www.lustre.org/ [12481.908294] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12481.935030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12483.438594] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12488.175575] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [12493.795068] LustreError: 331549:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12493.813624] LustreError: 331549:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12497.063673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12497.366889] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12502.093732] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [12509.785711] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12515.429063] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12523.021218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12523.608400] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12530.201562] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [12538.187464] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12552.178660] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12554.064877] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12558.820454] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12558.826452] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12558.845537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12558.858745] Lustre: Skipped 1 previous similar message [12558.870150] Lustre: Skipped 2 previous similar messages [12561.168798] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.60@tcp (stopping) [12563.941696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12563.948494] Lustre: Skipped 2 previous similar messages [12564.763658] Lustre: server umount lustre-MDT0000 complete [12566.236391] LustreError: 331545:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12568.931127] LustreError: 331529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787146639 with bad export cookie 1862664428086795917 [12568.940967] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12568.942730] LustreError: 331529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12569.063412] LustreError: 331546:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12569.064783] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12569.103169] Lustre: Skipped 1 previous similar message [12569.107849] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12571.358763] LustreError: 331546:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12576.480098] LustreError: 333903:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12576.482903] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.60@tcp (stopping) [12576.510370] LustreError: 333903:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12576.530873] Lustre: Skipped 2 previous similar messages [12583.391769] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12583.756850] Lustre: server umount lustre-MDT0001 complete [12588.144125] Lustre: server umount lustre-OST0000 complete [12598.482396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12604.683618] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [12609.331480] Lustre: 335733:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.160@tcp [12613.436267] Lustre: server umount MGS complete [12625.738760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12626.295659] LustreError: 336305:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12626.313697] LustreError: 336305:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [12626.403535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12632.284144] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [12643.045699] LustreError: 336302:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12643.069803] LustreError: 336302:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [12644.816585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12645.337099] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12650.974847] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [12664.060305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12664.589704] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12673.328855] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [12682.448140] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12683.859081] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12689.508271] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12691.353706] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12697.568341] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12697.577770] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12697.584073] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12697.602199] Lustre: Skipped 1 previous similar message [12697.613656] Lustre: Skipped 4 previous similar messages [12702.287880] Lustre: server umount lustre-OST0000 complete [12706.786858] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12706.799448] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12712.551480] Lustre: server umount lustre-MDT0000 complete [12712.934889] LustreError: 336301:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12712.961240] LustreError: 336301:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12716.900837] LustreError: 336285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787146787 with bad export cookie 1862664428086796757 [12716.906570] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12716.908636] LustreError: 336285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12717.271681] Lustre: server umount lustre-MDT0001 complete [12730.454908] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [12733.235459] Key type lgssc unregistered [12733.530506] LNet: 339748:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12733.540275] LNetError: 339748:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12733.568840] LNet: Removed LNI 192.168.204.160@tcp [12734.542478] Key type .llcrypt unregistered [12734.548162] Key type ._llcrypt unregistered [12752.987108] Key type ._llcrypt registered [12752.991727] Key type .llcrypt registered [12753.094943] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [12770.315290] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [12771.508058] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12771.617586] alg: No test for adler32 (adler32-zlib) [12772.790658] Lustre: Lustre: Build Version: 2.17.57_1_g9da61ce [12773.106557] LNet: Added LNI 192.168.204.160@tcp [8/256/0/180] [12774.791343] Key type lgssc registered [12776.111290] Lustre: Echo OBD driver; http://www.lustre.org/ [12818.892056] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12818.944127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12820.301426] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12820.339114] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12820.429858] Lustre: lustre-MDT0000: new disk, initializing [12820.506214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12820.522150] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12825.113625] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [12837.813601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12837.950477] Lustre: 342913:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12838.008568] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12838.012805] Lustre: Skipped 1 previous similar message [12838.137204] Lustre: lustre-MDT0001: new disk, initializing [12838.243450] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12838.307657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12838.319178] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12843.408995] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [12853.672562] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12859.697197] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12867.220260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12867.618148] Lustre: lustre-OST0000: new disk, initializing [12867.629092] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12867.636672] Lustre: 344880:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12867.805150] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12874.536987] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [12875.848461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12875.859568] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12876.042318] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12884.420675] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12889.476221] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12889.732629] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12894.221407] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12894.406268] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12896.737970] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12896.741184] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12896.757300] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12896.772185] Lustre: Skipped 1 previous similar message [12896.788579] Lustre: Skipped 1 previous similar message [12901.860149] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12901.868596] Lustre: Skipped 1 previous similar message [12902.505137] Lustre: server umount lustre-OST0000 complete [12906.984585] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12906.990513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12906.999048] Lustre: Skipped 1 previous similar message [12912.099395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12912.109257] Lustre: Skipped 2 previous similar messages [12912.350202] Lustre: server umount lustre-MDT0000 complete [12915.430441] LustreError: 342904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787146986 with bad export cookie 7140650880729360645 [12915.433774] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12915.446169] LustreError: 342904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12915.712044] Lustre: server umount lustre-MDT0001 complete [12922.710465] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 09:43:11 (1787146991) [12939.802707] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [12950.331645] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [12989.111030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12989.307073] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12989.352951] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12989.428980] Lustre: lustre-MDT0000: new disk, initializing [12989.515369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12989.530150] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12994.341634] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [13004.080598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13004.212836] Lustre: 349640:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13004.274642] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13004.279039] Lustre: Skipped 1 previous similar message [13004.373233] Lustre: lustre-MDT0001: new disk, initializing [13004.470944] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13004.541311] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13004.555522] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13008.628762] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [13016.959864] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13022.803821] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13030.783813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13031.122440] Lustre: lustre-OST0000: new disk, initializing [13031.125996] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13031.133956] Lustre: 351609:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13031.212452] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13038.205397] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [13039.189822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13039.204563] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13039.277144] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13047.538819] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13078.931476] Lustre: server umount lustre-OST0000 complete [13085.675712] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13085.688108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13085.692436] Lustre: Skipped 1 previous similar message [13092.924125] Lustre: server umount lustre-MDT0000 complete [13096.829768] LustreError: 349631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787147167 with bad export cookie 7140650880729361527 [13096.853172] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13107.420690] Lustre: server umount lustre-MDT0001 complete [13120.534163] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [13124.066994] Key type lgssc unregistered [13124.481715] LNet: 353768:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13124.502285] LNetError: 353768:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13124.536534] LNet: Removed LNI 192.168.204.160@tcp [13125.489585] Key type .llcrypt unregistered [13125.491881] Key type ._llcrypt unregistered [13219.176357] Key type ._llcrypt registered [13219.178555] Key type .llcrypt registered [13219.286182] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [13219.577350] Key type .llcrypt unregistered [13219.579904] Key type ._llcrypt unregistered [13300.096000] Key type ._llcrypt registered [13300.099693] Key type .llcrypt registered [13300.218295] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [13300.492838] Key type .llcrypt unregistered [13300.495783] Key type ._llcrypt unregistered [13327.961595] Key type ._llcrypt registered [13327.968447] Key type .llcrypt registered [13328.052390] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [13328.318970] Key type .llcrypt unregistered [13328.321663] Key type ._llcrypt unregistered [13358.093875] Key type ._llcrypt registered [13358.096409] Key type .llcrypt registered [13358.244544] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [13358.639530] Key type .llcrypt unregistered [13358.642530] Key type ._llcrypt unregistered [13431.531487] Key type ._llcrypt registered [13431.534381] Key type .llcrypt registered [13431.632487] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [13432.008280] Key type .llcrypt unregistered [13432.011083] Key type ._llcrypt unregistered [13460.809648] Key type ._llcrypt registered [13460.815925] Key type .llcrypt registered [13460.955785] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [13461.354761] Key type .llcrypt unregistered [13461.358699] Key type ._llcrypt unregistered [13491.550959] Key type ._llcrypt registered [13491.554744] Key type .llcrypt registered [13491.687571] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [13492.032734] Key type .llcrypt unregistered [13492.039196] Key type ._llcrypt unregistered [13525.303211] Key type ._llcrypt registered [13525.305582] Key type .llcrypt registered [13525.409128] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [13525.812319] Key type .llcrypt unregistered [13525.814153] Key type ._llcrypt unregistered [13559.052728] Key type ._llcrypt registered [13559.057260] Key type .llcrypt registered [13559.147706] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [13559.463699] Key type .llcrypt unregistered [13559.467705] Key type ._llcrypt unregistered [13587.823611] Key type ._llcrypt registered [13587.829325] Key type .llcrypt registered [13587.936147] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [13588.274419] Key type .llcrypt unregistered [13588.281237] Key type ._llcrypt unregistered [13658.359623] Key type ._llcrypt registered [13658.362907] Key type .llcrypt registered [13658.496681] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [13658.857723] Key type .llcrypt unregistered [13658.861666] Key type ._llcrypt unregistered [13689.603664] Key type ._llcrypt registered [13689.608508] Key type .llcrypt registered [13689.702297] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [13690.039318] Key type .llcrypt unregistered [13690.046875] Key type ._llcrypt unregistered [13721.521899] Key type ._llcrypt registered [13721.527810] Key type .llcrypt registered [13721.613263] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [13721.901578] Key type .llcrypt unregistered [13721.903349] Key type ._llcrypt unregistered [13782.218761] Key type ._llcrypt registered [13782.220964] Key type .llcrypt registered [13782.309271] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [13798.992468] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [13800.071576] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13800.291707] alg: No test for adler32 (adler32-zlib) [13801.324735] Lustre: Lustre: Build Version: 2.17.57_1_g9da61ce [13801.577938] LNet: Added LNI 192.168.204.160@tcp [8/256/0/180] [13803.231142] Key type lgssc registered [13804.519609] Lustre: Echo OBD driver; http://www.lustre.org/ [13841.273866] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13841.299203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13842.532179] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13842.555034] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13842.627826] Lustre: tst250fs-MDT0000: new disk, initializing [13842.691240] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13842.707336] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13847.199269] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [13858.054340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13858.203803] Lustre: 374106:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13858.251522] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13858.256614] Lustre: Skipped 1 previous similar message [13858.315993] Lustre: tst250fs-MDT0001: new disk, initializing [13858.408470] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13858.453645] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13858.468630] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13863.082153] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [13868.309914] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13875.969722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13876.230521] Lustre: tst250fs-OST0000: new disk, initializing [13876.235388] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13876.240947] Lustre: 375738:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13876.349270] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13879.411943] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13879.424426] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13879.574380] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13883.735380] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [13894.766272] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13894.967087] Lustre: tst250fs-OST0001: new disk, initializing [13894.973113] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13894.979898] Lustre: 376613:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13895.089946] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13901.559807] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [13904.988697] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13904.995723] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13905.116266] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13907.552521] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13910.083387] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [13910.094787] Lustre: Skipped 1 previous similar message [13935.081913] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13935.089380] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13935.116383] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13936.103323] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13936.110254] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13936.137973] Lustre: Skipped 1 previous similar message [13937.752914] Lustre: server umount tst250fs-MDT0000 complete [13941.774379] LustreError: 374100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787148012 with bad export cookie 5379599182203979260 [13941.783963] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13941.786056] LustreError: 374100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13942.167789] Lustre: server umount tst250fs-MDT0001 complete [13956.382899] Lustre: server umount tst250fs-OST0000 complete [13969.451081] Lustre: server umount tst250fs-OST0001 complete [13989.384704] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [13998.200229] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [14027.343436] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 10:01:36 (1787148096) [14029.087829] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [14047.941214] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [14061.094572] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [14098.933410] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13850 sec ========= 10:02:47 (1787148167) [14101.463451] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 10:02:49 (1787148169) === [14105.513387] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 10:02:53 (1787148173) === [14148.081987] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [14151.276421] Key type lgssc unregistered [14151.592852] LNet: 385771:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14151.599878] LNetError: 385771:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14151.618767] LNet: Removed LNI 192.168.204.160@tcp [14152.788205] Key type .llcrypt unregistered [14152.790275] Key type ._llcrypt unregistered