[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 463214361 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002377] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010124] LSM: Security Framework initializing [ 0.011049] Yama: becoming mindful. [ 0.012037] SELinux: Initializing. [ 0.013060] *** VALIDATE selinux *** [ 0.021102] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026697] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028101] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030004] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032111] *** VALIDATE tmpfs *** [ 0.033433] *** VALIDATE proc *** [ 0.034157] *** VALIDATE cgroup *** [ 0.035009] *** VALIDATE cgroup2 *** [ 0.036274] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038123] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040027] Spectre V2 : User space: Vulnerable [ 0.041010] Speculative Store Bypass: Vulnerable [ 0.043841] debug: unmapping init [mem 0xffffffff9d059000-0xffffffff9d060fff] [ 0.045250] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046676] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047026] ... version: 2 [ 0.048012] ... bit width: 48 [ 0.049016] ... generic registers: 4 [ 0.050011] ... value mask: 0000ffffffffffff [ 0.051014] ... max period: 00007fffffffffff [ 0.052013] ... fixed-purpose events: 3 [ 0.053010] ... event mask: 000000070000000f [ 0.054314] rcu: Hierarchical SRCU implementation. [ 0.056299] smp: Bringing up secondary CPUs ... [ 0.057604] x86: Booting SMP configuration: [ 0.058030] .... node #0, CPUs: #1 #2 #3 [ 0.061322] smp: Brought up 1 node, 4 CPUs [ 0.063010] smpboot: Max logical packages: 1 [ 0.064018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.129018] node 0 deferred pages initialised in 62ms [ 0.131543] devtmpfs: initialized [ 0.132171] x86/mm: Memory block size: 128MB [ 0.134436] gcov: version magic: 0x41383552 [ 0.136334] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.138074] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.140228] pinctrl core: initialized pinctrl subsystem [ 0.143221] [ 0.143732] ************************************************************* [ 0.146028] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.150012] ** ** [ 0.154013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.156011] ** ** [ 0.159026] ** This means that this kernel is built to expose internal ** [ 0.162014] ** IOMMU data structures, which may compromise security on ** [ 0.164011] ** your system. ** [ 0.167013] ** ** [ 0.170011] ** If you see this message and you are not debugging the ** [ 0.172011] ** kernel, report this immediately to your vendor! ** [ 0.175013] ** ** [ 0.178014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.181012] ************************************************************* [ 0.184644] NET: Registered protocol family 16 [ 0.186460] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.189055] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.192061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.196095] cpuidle: using governor menu [ 0.197836] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.200594] PCI: Using configuration type 1 for base access [ 0.204128] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.217171] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.220140] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.225132] cryptd: max_cpu_qlen set to 1000 [ 0.228430] ACPI: Added _OSI(Module Device) [ 0.232015] ACPI: Added _OSI(Processor Device) [ 0.233014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.235013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.240000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.246550] ACPI: Interpreter enabled [ 0.248063] ACPI: PM: (supports S0 S3 S4 S5) [ 0.249014] ACPI: Using IOAPIC for interrupt routing [ 0.251090] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.255417] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.265824] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.268034] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.271019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.274073] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.279746] acpiphp: Slot [2] registered [ 0.281146] acpiphp: Slot [5] registered [ 0.283117] acpiphp: Slot [6] registered [ 0.284105] acpiphp: Slot [7] registered [ 0.285063] acpiphp: Slot [8] registered [ 0.287114] acpiphp: Slot [9] registered [ 0.288090] acpiphp: Slot [10] registered [ 0.290105] acpiphp: Slot [3] registered [ 0.291082] acpiphp: Slot [4] registered [ 0.292091] acpiphp: Slot [11] registered [ 0.294119] acpiphp: Slot [12] registered [ 0.295094] acpiphp: Slot [13] registered [ 0.297097] acpiphp: Slot [14] registered [ 0.298085] acpiphp: Slot [15] registered [ 0.300100] acpiphp: Slot [16] registered [ 0.301088] acpiphp: Slot [17] registered [ 0.303088] acpiphp: Slot [18] registered [ 0.305095] acpiphp: Slot [19] registered [ 0.306064] acpiphp: Slot [20] registered [ 0.307084] acpiphp: Slot [21] registered [ 0.308076] acpiphp: Slot [22] registered [ 0.310097] acpiphp: Slot [23] registered [ 0.312102] acpiphp: Slot [24] registered [ 0.313089] acpiphp: Slot [25] registered [ 0.315088] acpiphp: Slot [26] registered [ 0.316111] acpiphp: Slot [27] registered [ 0.318086] acpiphp: Slot [28] registered [ 0.319072] acpiphp: Slot [29] registered [ 0.321221] acpiphp: Slot [30] registered [ 0.322091] acpiphp: Slot [31] registered [ 0.324127] PCI host bridge to bus 0000:00 [ 0.325018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.327020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.330022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.332026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.335024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.339134] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.342297] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.344961] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.349474] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.359751] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.365729] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.369034] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.371021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.376019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.380011] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.382695] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.386049] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.390059] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.397015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.414013] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.418014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.425265] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.432015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.438025] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.456016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.465531] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.478014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.492020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.517017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.530573] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.544015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.554014] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.576016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.588424] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.600016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.609014] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.625016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.635933] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.641012] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.650025] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.670027] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.680360] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.691015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.704014] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.725016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.736000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.740438] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.742462] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.745402] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.747292] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.752750] iommu: Default domain type: Passthrough [ 0.754480] SCSI subsystem initialized [ 0.756141] ACPI: bus type USB registered [ 0.757131] usbcore: registered new interface driver usbfs [ 0.759083] usbcore: registered new interface driver hub [ 0.761083] usbcore: registered new device driver usb [ 0.763214] pps_core: LinuxPPS API ver. 1 registered [ 0.765015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.770114] PTP clock support registered [ 0.772169] EDAC MC: Ver: 3.0.0 [ 0.773514] PCI: Using ACPI for IRQ routing [ 0.775514] NetLabel: Initializing [ 0.777012] NetLabel: domain hash size = 128 [ 0.779011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.781079] NetLabel: unlabeled traffic allowed by default [ 0.784041] vgaarb: loaded [ 0.785336] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.787018] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.793005] clocksource: Switched to clocksource kvm-clock [ 0.904167] VFS: Disk quotas dquot_6.6.0 [ 0.905873] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.909750] *** VALIDATE ramfs *** [ 0.911145] *** VALIDATE hugetlbfs *** [ 0.912775] pnp: PnP ACPI init [ 0.915331] pnp: PnP ACPI: found 6 devices [ 0.935819] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.940460] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.942292] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.944630] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.947023] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.950043] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.953651] NET: Registered protocol family 2 [ 0.956543] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.961598] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.965307] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.969950] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.973163] TCP: Hash tables configured (established 65536 bind 65536) [ 0.976584] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.979813] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.982639] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.985934] NET: Registered protocol family 1 [ 0.988812] RPC: Registered named UNIX socket transport module. [ 0.991111] RPC: Registered udp transport module. [ 0.992920] RPC: Registered tcp transport module. [ 0.994762] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.997399] NET: Registered protocol family 44 [ 0.999403] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.001572] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.004082] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.006876] PCI: CLS 0 bytes, default 64 [ 1.009571] Unpacking initramfs... [ 2.407986] debug: unmapping init [mem 0xffff9fa27cc54000-0xffff9fa27ffbffff] [ 2.411419] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.414087] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.417540] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.935052] Initialise system trusted keyrings [ 2.936706] Key type blacklist registered [ 2.939209] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.953977] zbud: loaded [ 2.957972] *** VALIDATE nfs *** [ 2.959454] *** VALIDATE nfs4 *** [ 2.960994] pstore: using deflate compression [ 2.964570] Platform Keyring initialized [ 3.078216] NET: Registered protocol family 38 [ 3.080274] Key type asymmetric registered [ 3.082101] Asymmetric key parser 'x509' registered [ 3.084111] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.086516] io scheduler mq-deadline registered [ 3.088374] io scheduler kyber registered [ 3.090163] io scheduler bfq registered [ 3.092594] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.096349] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.100042] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.103442] ACPI: Power Button [PWRF] [ 3.109486] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.117237] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.129597] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.137099] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.153773] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.181914] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.212116] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.216439] Non-volatile memory driver v1.3 [ 3.217779] Linux agpgart interface v0.103 [ 3.255959] virtio_blk virtio1: [vda] 146152 512-byte logical blocks (74.8 MB/71.4 MiB) [ 3.259222] vda: detected capacity change from 0 to 74829824 [ 3.284541] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.287189] vdb: detected capacity change from 0 to 1073741824 [ 3.302208] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.305280] vdc: detected capacity change from 0 to 2621440000 [ 3.321861] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.325308] vdd: detected capacity change from 0 to 2621440000 [ 3.340577] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.343562] vde: detected capacity change from 0 to 4294967296 [ 3.364083] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.366686] vdf: detected capacity change from 0 to 4294967296 [ 3.382933] libphy: Fixed MDIO Bus: probed [ 3.395672] usbcore: registered new interface driver usbserial_generic [ 3.397687] usbserial: USB Serial support registered for generic [ 3.399419] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.402942] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.404355] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.406367] mousedev: PS/2 mouse device common for all mice [ 3.408852] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.411553] rtc_cmos 00:05: RTC can wake from S4 [ 3.413554] rtc_cmos 00:05: registered as rtc0 [ 3.415843] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.418298] intel_pstate: CPU model not supported [ 3.419331] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.426037] hid: raw HID events driver (C) Jiri Kosina [ 3.428691] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.429191] usbcore: registered new interface driver usbhid [ 3.433447] usbhid: USB HID core driver [ 3.435227] drop_monitor: Initializing network drop monitor service [ 3.437295] Initializing XFRM netlink socket [ 3.438861] NET: Registered protocol family 10 [ 3.442913] Segment Routing with IPv6 [ 3.444479] NET: Registered protocol family 17 [ 3.446246] mpls_gso: MPLS GSO support [ 3.451658] RAS: Correctable Errors collector initialized. [ 3.453890] AVX version of gcm_enc/dec engaged. [ 3.455481] AES CTR mode by8 optimization enabled [ 3.524940] sched_clock: Marking stable (3524921034, 0)->(4420971600, -896050566) [ 3.527607] registered taskstats version 1 [ 3.529365] Loading compiled-in X.509 certificates [ 3.530981] zswap: loaded using pool lzo/zbud [ 3.553390] Key type big_key registered [ 3.565765] Key type encrypted registered [ 3.567318] ima: No TPM chip found, activating TPM-bypass! [ 3.569505] ima: Allocated hash algorithm: sha1 [ 3.571209] ima: No architecture policies found [ 3.572808] evm: Initialising EVM extended attributes: [ 3.574487] evm: security.selinux [ 3.575738] evm: security.ima [ 3.576975] evm: security.capability [ 3.578396] evm: HMAC attrs: 0x1 [ 3.580633] rtc_cmos 00:05: setting system clock to 2026-08-27 12:07:40 UTC (1787832460) [ 3.586822] debug: unmapping init [mem 0xffffffff9e003000-0xffffffff9e1fffff] [ 3.590027] debug: unmapping init [mem 0xffffffff9cd82000-0xffffffff9d058fff] [ 3.596091] Write protecting the kernel read-only data: 28672k [ 3.599478] debug: unmapping init [mem 0xffffffff9b403000-0xffffffff9b5fffff] [ 3.602172] debug: unmapping init [mem 0xffffffff9bd14000-0xffffffff9bdfffff] [ 3.632502] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.640626] systemd[1]: Detected virtualization kvm. [ 3.642522] systemd[1]: Detected architecture x86-64. [ 3.644670] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.670258] systemd[1]: No hostname configured. [ 3.671859] systemd[1]: Set hostname to . [ 3.674093] random: systemd: uninitialized urandom read (16 bytes read) [ 3.676443] systemd[1]: Initializing machine ID from random generator. [ 3.718201] random: ln: uninitialized urandom read (6 bytes read) [ 3.796778] random: systemd: uninitialized urandom read (16 bytes read) [ 3.800194] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.807672] systemd[1]: Started Memstrack Anylazing Service. [ OK ] Started Memstrack Anylazing Service. [ 3.816390] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Setup Virtual Console... [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.400646] device-mapper: uevent: version 1.0.3 [ 4.402965] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.057912] virtio_net virtio0 ens2: renamed from eth0 [ 5.076160] scsi host0: ata_piix [ 5.090695] random: fast init done [ 5.105496] scsi host1: ata_piix [ 5.107738] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.111135] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.302701] dracut-initqueue[586]: RTNETLINK answers: File exists [ 10.084055] random: crng init done [ 10.085536] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.313345] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.364433] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.590307] SELinux: Disabled at runtime. [ 11.646950] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.658779] systemd[1]: Detected virtualization kvm. [ 11.661462] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.095131] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.097596] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.102072] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.106317] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.110181] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.119783] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.129522] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target RPC Port Mapper. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Control Socket. Mounting Kernel Debug File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target rpc_pipefs.target. [ 12.172898] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Huge Pages File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice User and Session Slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. Starting Apply Kernel Variables... [ OK ] Created slice system-serial\x2dgetty.slice. Mounting POSIX Message Queue File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd Root File System. Starting Remount Root and Kernel File Systems... [ OK ] Reached target Slices. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 12.525111] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 12.834293] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.913792] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.988351] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.997834] EDAC sbridge: Ver: 1.1.2 [ 14.212304] Key type dns_resolver registered [ 14.520626] NFS: Registering the id_resolver key type [ 14.522561] Key type id_resolver registered [ 14.524088] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg608-server login: [ 62.962993] libcfs: loading out-of-tree module taints kernel. [ 63.000384] Key type ._llcrypt registered [ 63.003310] Key type .llcrypt registered [ 63.082416] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 83.254365] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 84.744647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 84.767671] alg: No test for adler32 (adler32-zlib) [ 86.168240] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 87.005505] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 88.728548] Key type lgssc registered [ 90.560368] Lustre: Echo OBD driver; http://www.lustre.org/ [ 108.240957] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 120.931178] hrtimer: interrupt took 5231857 ns [ 147.340620] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 159.332068] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 159.343685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 160.549768] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 160.576202] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 160.635467] Lustre: lustre-MDT0000: new disk, initializing [ 160.770991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 160.811634] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 165.085323] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 178.748161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 178.885926] Lustre: 6513:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 178.910892] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 178.917867] Lustre: Skipped 1 previous similar message [ 179.019885] Lustre: lustre-MDT0001: new disk, initializing [ 179.146327] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 179.199923] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 179.210552] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 184.453032] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 189.059956] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 197.581726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 197.821806] Lustre: lustre-OST0000: new disk, initializing [ 197.826746] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 197.833493] Lustre: 8448:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 197.904652] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 202.793826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 202.802659] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 202.876852] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 203.957419] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 216.915613] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 217.014779] Lustre: lustre-OST0001: new disk, initializing [ 217.018358] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 217.025351] Lustre: 9519:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 217.080335] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 222.562689] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 226.323328] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 226.336673] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 226.418200] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 234.156769] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 241.504532] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 247.388270] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing check_logdir /tmp/testlogs/ [ 251.743421] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing yml_node [ 257.086197] Lustre: DEBUG MARKER: Client: 2.17.57.80 [ 259.809895] Lustre: DEBUG MARKER: MDS: 2.17.57.80 [ 263.030448] Lustre: DEBUG MARKER: OSS: 2.17.57.80 [ 264.793961] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 08:12:01 EDT 2026 [ 280.794322] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 282.272439] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 297.955435] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 297.958797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 297.972189] Lustre: Skipped 1 previous similar message [ 303.072897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 303.087440] Lustre: Skipped 6 previous similar messages [ 303.430628] Lustre: server umount lustre-MDT0000 complete [ 306.918213] LustreError: 6505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832763 with bad export cookie 7101889254696214004 [ 306.928403] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 306.929178] LustreError: 6505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 308.196642] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 308.197255] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 308.209201] Lustre: Skipped 3 previous similar messages [ 311.200085] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 311.203840] Lustre: Skipped 2 previous similar messages [ 313.352563] Lustre: server umount lustre-MDT0001 complete [ 316.762455] Lustre: server umount lustre-OST0000 complete [ 320.448514] Lustre: server umount lustre-OST0001 complete [ 326.570711] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 336.361903] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 372.168658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 372.695412] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 372.741557] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 372.910397] Lustre: lustre-MDT0000: new disk, initializing [ 373.064242] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 373.089820] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 378.986126] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 389.671875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 389.794463] Lustre: 15073:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 389.871371] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 389.875780] Lustre: Skipped 1 previous similar message [ 389.971546] Lustre: lustre-MDT0001: new disk, initializing [ 390.071579] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 390.125890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 390.146749] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 394.028603] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 401.734508] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 407.118556] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 413.336170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 413.586942] Lustre: lustre-OST0000: new disk, initializing [ 413.590746] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 413.597185] Lustre: 17045:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 413.687291] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 415.500657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 415.526113] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 415.735169] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 419.989410] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 427.748282] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 431.060312] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 431.252493] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 435.110688] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 435.338884] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 441.324391] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 441.329702] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 441.348390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 441.358854] Lustre: Skipped 1 previous similar message [ 441.376483] Lustre: Skipped 1 previous similar message [ 443.479132] Lustre: server umount lustre-OST0000 complete [ 451.555580] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 451.558135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 451.578467] Lustre: Skipped 1 previous similar message [ 451.579657] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 451.582898] Lustre: Skipped 1 previous similar message [ 451.598066] LustreError: Skipped 1 previous similar message [ 452.829861] Lustre: server umount lustre-MDT0000 complete [ 456.478275] LustreError: 15064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832913 with bad export cookie 7101889254696215537 [ 456.482330] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 456.486060] LustreError: 15064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 456.851614] Lustre: server umount lustre-MDT0001 complete [ 462.843978] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 08:15:19 (1787832919) [ 478.603777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 479.009966] LustreError: 19917:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 479.031015] LustreError: 19917:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 479.109254] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 483.321959] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 484.333245] LustreError: 19918:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 488.428462] LustreError: 19917:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 491.490246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 496.128373] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 503.294974] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 508.324890] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 510.111153] Lustre: Failing over lustre-MDT0000 [ 510.328782] Lustre: server umount lustre-MDT0000 complete [ 512.480374] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 512.489574] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 512.508878] LustreError: 19918:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 516.576307] LustreError: 19918:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 516.587818] LustreError: 19918:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 526.819665] LustreError: 19917:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 526.837508] LustreError: 19917:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 528.373509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 528.519301] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 533.101900] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 533.992906] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 533.998368] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 534.026134] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 542.096919] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 548.324379] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 548.338577] Lustre: Skipped 1 previous similar message [ 548.349309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 548.361259] Lustre: Skipped 1 previous similar message [ 552.707380] Lustre: server umount lustre-MDT0000 complete [ 553.443480] LustreError: 19917:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 553.459512] LustreError: 19917:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 556.567391] LustreError: 19897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833013 with bad export cookie 7101889254696216951 [ 556.574026] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 556.576246] LustreError: 19897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 556.978821] Lustre: server umount lustre-MDT0001 complete [ 563.958168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 564.547512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 564.560398] Lustre: Skipped 2 previous similar messages [ 568.716339] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 576.385038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 581.193146] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 588.807865] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 593.635695] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 595.512040] Lustre: Failing over lustre-MDT0000 [ 595.732629] Lustre: server umount lustre-MDT0000 complete [ 597.474448] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 597.482856] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 597.507463] LustreError: 24273:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 597.524595] LustreError: 24273:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 613.264447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 613.423432] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 613.777092] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 617.473052] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 618.989624] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 618.995252] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 619.005757] Lustre: Skipped 1 previous similar message [ 619.025696] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 625.118871] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 629.216810] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 629.228482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 633.315954] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 633.328397] Lustre: Skipped 2 previous similar messages [ 634.963263] Lustre: server umount lustre-MDT0000 complete [ 638.305260] LustreError: 23233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833095 with bad export cookie 7101889254696217889 [ 638.313778] LustreError: 23233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 638.315582] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 638.593905] Lustre: server umount lustre-MDT0001 complete [ 648.051540] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 08:18:23 (1787833103) [ 663.475261] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 671.005721] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 707.521444] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 717.139333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 717.319886] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 717.378681] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 717.439915] Lustre: lustre-MDT0000: new disk, initializing [ 717.491379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 717.494061] Lustre: Skipped 2 previous similar messages [ 717.505106] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 721.590234] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 731.001568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 731.134260] Lustre: 29622:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 731.430097] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 735.568763] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 743.743164] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 748.539724] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 754.583243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 754.803638] Lustre: 31589:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 756.061721] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 756.110572] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 756.168060] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 756.191664] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 760.397577] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 768.579351] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 773.404786] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 792.046693] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 792.047997] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 792.056183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 792.065028] Lustre: Skipped 1 previous similar message [ 792.084078] Lustre: Skipped 3 previous similar messages [ 795.500132] Lustre: server umount lustre-OST0000 complete [ 805.122346] Lustre: server umount lustre-MDT0000 complete [ 807.397769] LustreError: 29629:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 807.412907] LustreError: 29629:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 808.167237] LustreError: 32526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833265 with bad export cookie 7101889254696218295 [ 808.168154] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 808.177390] LustreError: 32526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 808.577384] Lustre: server umount lustre-MDT0001 complete [ 818.260224] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 820.986561] Key type lgssc unregistered [ 821.361477] LNet: 33567:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 821.372929] LNetError: 33567:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 821.389867] LNet: Removed LNI 192.168.206.108@tcp [ 822.339381] Key type .llcrypt unregistered [ 822.342873] Key type ._llcrypt unregistered [ 846.342341] Key type ._llcrypt registered [ 846.344520] Key type .llcrypt registered [ 846.418661] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 847.127286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 847.172895] alg: No test for adler32 (adler32-zlib) [ 848.144530] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 848.367481] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 850.007333] Key type lgssc registered [ 850.803819] Lustre: Echo OBD driver; http://www.lustre.org/ [ 857.431492] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 857.447132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 858.962751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 863.764578] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 868.330879] LustreError: 34980:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 868.359234] LustreError: 34980:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 872.107801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 872.457299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 877.229593] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 883.715399] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 888.692220] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 894.811471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 895.114733] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 897.156986] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 900.468866] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 903.293297] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 907.096993] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 919.931725] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 08:22:56 (1787833376) [ 921.413552] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 922.968226] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 928.542624] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 08:23:05 (1787833385) [ 935.924776] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 08:23:12 (1787833392) [ 936.991932] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 944.409679] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 944.417225] Lustre: Skipped 2 previous similar messages [ 945.943451] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 08:23:22 (1787833402) [ 948.767375] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 948.772486] Lustre: Skipped 3 previous similar messages [ 963.787898] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 963.794744] Lustre: Skipped 6 previous similar messages [ 966.089536] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 08:23:42 (1787833422) [ 968.582479] Lustre: *** cfs_fail_loc=131b, val=2*** [ 972.013480] Lustre: *** cfs_fail_loc=131b, val=2*** [ 979.563981] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 08:23:56 (1787833436) [ 981.145535] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 981.161532] Lustre: Skipped 3 previous similar messages [ 991.571417] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 08:24:08 (1787833448) [ 1006.048394] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1006.055159] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1006.074596] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1006.105037] Lustre: Skipped 1 previous similar message [ 1011.181431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1011.192496] Lustre: Skipped 2 previous similar messages [ 1011.639363] Lustre: server umount lustre-OST0000 complete [ 1015.776785] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1015.788699] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1015.801712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1021.411100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1021.421507] Lustre: Skipped 3 previous similar messages [ 1021.836804] Lustre: server umount lustre-MDT0000 complete [ 1025.171217] LustreError: 34960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833482 with bad export cookie 6952938813402044661 [ 1025.174426] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1025.184831] LustreError: 34960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1025.495293] Lustre: server umount lustre-MDT0001 complete [ 1035.322701] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 1037.750488] Key type lgssc unregistered [ 1037.966266] LNet: 42835:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1037.975481] LNetError: 42835:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1037.992798] LNet: Removed LNI 192.168.206.108@tcp [ 1038.650234] Key type .llcrypt unregistered [ 1038.654336] Key type ._llcrypt unregistered [ 1051.967867] Key type ._llcrypt registered [ 1051.970602] Key type .llcrypt registered [ 1052.078222] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 1065.925437] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 1066.797686] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1066.891800] alg: No test for adler32 (adler32-zlib) [ 1067.966330] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 1068.243260] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 1069.935157] Key type lgssc registered [ 1070.954490] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1100.330094] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1100.378341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1101.583846] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1101.608519] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1101.709742] Lustre: lustre-MDT0000: new disk, initializing [ 1101.809366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1101.826060] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1106.127440] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1115.501196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1115.593529] Lustre: 45984:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1115.641297] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1115.645771] Lustre: Skipped 1 previous similar message [ 1115.713526] Lustre: lustre-MDT0001: new disk, initializing [ 1115.791718] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1115.850302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1115.859528] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1119.962153] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1127.932906] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1132.900673] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1138.017374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1138.241303] Lustre: lustre-OST0000: new disk, initializing [ 1138.244491] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1138.249478] Lustre: 47953:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1138.318221] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1143.190959] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1148.491554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1148.502565] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1148.605832] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1152.044469] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1156.404650] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1156.630580] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1162.133263] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1162.341794] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1164.256817] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1164.259544] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1164.262686] LustreError: Skipped 1 previous similar message [ 1164.263488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1164.276494] Lustre: Skipped 1 previous similar message [ 1169.378335] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1169.384974] Lustre: Skipped 2 previous similar messages [ 1170.223729] Lustre: server umount lustre-OST0000 complete [ 1174.500321] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1174.516508] Lustre: Skipped 1 previous similar message [ 1174.521153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1174.526888] Lustre: Skipped 1 previous similar message [ 1179.512398] Lustre: server umount lustre-MDT0000 complete [ 1183.010020] LustreError: 45977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833639 with bad export cookie 13220858777732298425 [ 1183.016616] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1183.021371] LustreError: 45977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1183.254547] Lustre: server umount lustre-MDT0001 complete [ 1191.970714] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 08:27:28 (1787833648) [ 1197.642876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1198.122847] LustreError: 50412:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1198.150955] LustreError: 50412:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1198.225204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1202.854869] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1203.681382] LustreError: 50413:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1208.802532] LustreError: 50412:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1211.648652] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1212.092823] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1216.675144] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1223.256416] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1228.620910] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1235.721266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1236.003496] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1242.177434] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1250.287426] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1254.874164] LustreError: 52942:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1255.948428] Lustre: Modifying parameter general.timeout=1 in log params [ 1256.949499] Lustre: Modifying parameter general.timeout=2 in log params [ 1259.671766] Lustre: Modifying parameter general.timeout=5 in log params [ 1259.675494] Lustre: Skipped 2 previous similar messages [ 1264.406053] Lustre: Modifying parameter general.timeout=11 in log params [ 1264.409665] Lustre: Skipped 5 previous similar messages [ 1272.842835] Lustre: Modifying parameter general.timeout=21 in log params [ 1272.846343] Lustre: Skipped 9 previous similar messages [ 1289.665219] Lustre: Modifying parameter general.timeout=40 in log params [ 1289.670595] Lustre: Skipped 18 previous similar messages [ 1322.140192] Lustre: Modifying parameter general.timeout=78 in log params [ 1322.143618] Lustre: Skipped 37 previous similar messages [ 1374.573592] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 08:30:31 (1787833831) [ 1377.593324] Lustre: 59875:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1379.123321] Lustre: 59973:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1381.396575] Lustre: 60072:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1385.679303] Lustre: 60286:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1385.690794] Lustre: 60286:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1386.694501] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=5000 in log params [ 1386.708059] Lustre: Skipped 58 previous similar messages [ 1411.927967] Lustre: 61757:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1411.939915] Lustre: 61757:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1411.963266] LustreError: 61757:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1416.054477] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 08:31:12 (1787833872) [ 1428.961954] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1428.967316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1428.971683] Lustre: Skipped 1 previous similar message [ 1428.987836] Lustre: Skipped 2 previous similar messages [ 1433.782888] Lustre: server umount lustre-MDT0000 complete [ 1434.084525] LustreError: 51140:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1434.115388] LustreError: 51140:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1437.448605] LustreError: 50394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833894 with bad export cookie 13220858777732299300 [ 1437.457182] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1437.472856] LustreError: 50394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1438.110657] Lustre: server umount lustre-MDT0001 complete [ 1452.278134] Lustre: server umount lustre-OST0000 complete [ 1477.583992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1477.886529] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1478.181320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1482.511780] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1490.380697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1490.439920] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1490.462185] Lustre: 64283:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1494.804742] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1502.709871] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1508.352815] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1515.055404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1515.294307] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1515.637376] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1515.646359] Lustre: Skipped 1 previous similar message [ 1522.173101] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1528.842293] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1531.031729] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1531.042919] Lustre: Skipped 16 previous similar messages [ 1531.102644] Lustre: 66848:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1540.579322] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1540.589350] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1540.605459] Lustre: Skipped 1 previous similar message [ 1540.612896] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1544.409450] Lustre: server umount lustre-OST0000 complete [ 1551.328596] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1551.347827] Lustre: Skipped 1 previous similar message [ 1551.353794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1551.360073] Lustre: Skipped 3 previous similar messages [ 1554.165746] Lustre: server umount lustre-MDT0000 complete [ 1556.453280] LustreError: 64292:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1556.475365] LustreError: 64292:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1557.725373] LustreError: 64277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834014 with bad export cookie 13220858777732302233 [ 1557.729399] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1557.737716] LustreError: 64277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1558.075853] Lustre: server umount lustre-MDT0001 complete [ 1568.507904] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 1571.507800] Key type lgssc unregistered [ 1571.894768] LNet: 68059:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1571.908228] LNetError: 68059:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1571.920774] LNet: Removed LNI 192.168.206.108@tcp [ 1572.803079] Key type .llcrypt unregistered [ 1572.808479] Key type ._llcrypt unregistered [ 1598.438030] Key type ._llcrypt registered [ 1598.439862] Key type .llcrypt registered [ 1598.586030] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 1599.751577] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1599.844860] alg: No test for adler32 (adler32-zlib) [ 1600.965539] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 1601.273641] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 1602.935181] Key type lgssc registered [ 1604.043696] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1611.790228] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1611.805162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1613.502385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1618.659538] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1624.040554] LustreError: 69474:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1624.063527] LustreError: 69474:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1627.425868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1627.908567] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1632.108286] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1638.994963] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1644.240370] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1650.749763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1651.320411] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1657.480345] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1664.708468] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1673.219094] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1683.936593] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1683.946573] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1683.956986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1684.450400] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1684.458741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1684.466747] Lustre: Skipped 1 previous similar message [ 1684.477244] Lustre: Skipped 2 previous similar messages [ 1687.056564] Lustre: server umount lustre-MDT0000 complete [ 1689.571770] LustreError: 70532:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1689.607489] LustreError: 70532:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1690.481614] LustreError: 69455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834147 with bad export cookie 699236774659845032 [ 1690.485623] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1690.492904] LustreError: 69455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1690.819505] Lustre: server umount lustre-MDT0001 complete [ 1704.964465] Lustre: server umount lustre-OST0000 complete [ 1729.218172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1729.472552] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1729.525370] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1729.807861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1734.319285] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1742.546069] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1742.639661] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1742.702974] Lustre: 74347:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1743.282543] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1747.519802] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1754.639419] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1759.734299] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1766.092048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1766.307819] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1766.726017] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1772.715607] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1779.272885] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1780.798362] Lustre: 76921:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1782.443448] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1788.389278] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1788.399377] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1788.414847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1793.328669] Lustre: server umount lustre-OST0000 complete [ 1799.136102] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1799.140073] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1799.153417] LustreError: Skipped 1 previous similar message [ 1799.154244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1799.154248] Lustre: Skipped 1 previous similar message [ 1799.179417] Lustre: Skipped 2 previous similar messages [ 1803.008753] Lustre: server umount lustre-MDT0000 complete [ 1804.258395] LustreError: 75668:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1804.271146] LustreError: 75668:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1806.284412] LustreError: 74338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834263 with bad export cookie 699236774659845865 [ 1806.286120] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1806.296975] LustreError: 74338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1806.643760] Lustre: server umount lustre-MDT0001 complete [ 1816.341375] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 1819.013128] Key type lgssc unregistered [ 1819.286539] LNet: 78139:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1819.297660] LNetError: 78139:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1819.312772] LNet: Removed LNI 192.168.206.108@tcp [ 1820.115629] Key type .llcrypt unregistered [ 1820.117463] Key type ._llcrypt unregistered [ 1841.352970] Key type ._llcrypt registered [ 1841.354227] Key type .llcrypt registered [ 1841.414831] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 1842.342894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1842.482684] alg: No test for adler32 (adler32-zlib) [ 1843.567232] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 1843.845337] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 1845.515165] Key type lgssc registered [ 1846.590131] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1854.644386] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1854.660590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1856.293533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1860.777623] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1866.721044] LustreError: 79551:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1866.734040] LustreError: 79551:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1870.233103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1870.963736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1874.851692] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1881.474720] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1886.144554] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1891.953069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1892.499536] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1898.307845] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1905.163856] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1910.162839] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1910.825974] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1912.328808] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1912.335689] Lustre: Skipped 1 previous similar message [ 1914.439684] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1914.446480] Lustre: Skipped 2 previous similar messages [ 1919.036406] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1919.039791] Lustre: Skipped 5 previous similar messages [ 1927.755142] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1927.759847] Lustre: Skipped 12 previous similar messages [ 1944.034359] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1944.044603] Lustre: Skipped 19 previous similar messages [ 1976.240843] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1976.248485] Lustre: Skipped 38 previous similar messages [ 1996.990648] Lustre: 87645:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1999.313324] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 08:40:56 (1787834456) [ 2000.689188] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2002.019262] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 08:40:59 (1787834459) [ 2008.544685] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2008.557300] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2008.598482] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2009.568659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2009.572871] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2009.575669] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2009.582424] Lustre: Skipped 1 previous similar message [ 2012.002772] Lustre: server umount lustre-OST0000 complete [ 2019.296973] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2019.311706] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2019.340324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2022.760717] Lustre: server umount lustre-MDT0000 complete [ 2024.939966] LustreError: 79570:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2025.895098] LustreError: 79532:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834482 with bad export cookie 15972764063592501690 [ 2025.903145] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2025.907812] LustreError: 79532:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2026.161921] Lustre: server umount lustre-MDT0001 complete [ 2035.899284] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 2038.362604] Key type lgssc unregistered [ 2038.724749] LNet: 88899:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2038.733288] LNetError: 88899:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2038.759626] LNet: Removed LNI 192.168.206.108@tcp [ 2039.570419] Key type .llcrypt unregistered [ 2039.572304] Key type ._llcrypt unregistered [ 2043.736664] Key type ._llcrypt registered [ 2043.739101] Key type .llcrypt registered [ 2043.858917] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2047.882376] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules [ 2049.296451] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2049.343289] alg: No test for adler32 (adler32-zlib) [ 2050.388666] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 2050.646585] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 2052.327176] Key type lgssc registered [ 2053.487129] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2054.765886] LustreError: 90104:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2072.287111] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 2076.122606] LustreError: 90104:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2080.451932] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2080.476066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2081.847822] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2085.362652] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2092.001473] LustreError: 91126:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2092.026921] LustreError: 91126:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2092.741980] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 08:42:29 (1787834549) [ 2096.488714] LustreError: 91158:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2096.496836] LustreError: 91158:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2096.502386] LustreError: 91158:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 15, retries 0, failed: rc = -5 [ 2096.823351] Lustre: server umount lustre-MDT0000 complete [ 2108.612688] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 2111.084295] Key type lgssc unregistered [ 2111.312778] LNet: 92652:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2111.318707] LNetError: 92652:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2111.344181] LNet: Removed LNI 192.168.206.108@tcp [ 2112.147203] Key type .llcrypt unregistered [ 2112.151513] Key type ._llcrypt unregistered [ 2130.376170] Key type ._llcrypt registered [ 2130.383855] Key type .llcrypt registered [ 2130.502140] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 2131.942803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2131.994775] alg: No test for adler32 (adler32-zlib) [ 2133.027219] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 2133.290728] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 2134.991224] Key type lgssc registered [ 2136.121941] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2143.234840] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2143.263829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2144.716620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2149.046610] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2154.981043] LustreError: 93872:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2155.002506] LustreError: 93872:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2157.268788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2157.983100] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2162.602916] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2169.813981] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2175.325433] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2181.376803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2181.991201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2188.225679] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2195.644995] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2242.053340] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 08:44:58 (1787834698) [ 2243.531312] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2244.976761] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 08:45:01 (1787834701) [ 2249.710768] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2249.717379] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2249.726432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2250.728659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2250.729903] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2250.746725] Lustre: Skipped 1 previous similar message [ 2254.553942] Lustre: server umount lustre-MDT0000 complete [ 2255.842412] LustreError: 94603:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2255.861036] LustreError: 94603:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2257.761690] LustreError: 93852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834714 with bad export cookie 1164316001551252180 [ 2257.764722] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2257.771991] LustreError: 93852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2258.102720] Lustre: server umount lustre-MDT0001 complete [ 2272.222757] Lustre: server umount lustre-OST0000 complete [ 2282.036905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2282.385455] LustreError: 98148:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2282.527649] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2287.102987] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2287.589063] LustreError: 98149:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2292.714771] LustreError: 98148:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2295.923745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2296.899629] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2301.635846] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2308.357895] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2313.729355] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2325.972612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2326.194768] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2326.199534] LustreError: 98137:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2326.218808] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2326.226496] LustreError: 100231:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2326.230993] LustreError: 100231:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2326.235796] LustreError: 100231:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2326.292992] Lustre: server umount lustre-OST0000 complete [ 2326.299372] LustreError: 100231:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2334.731524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2335.000718] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2335.012627] LustreError: 98136:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2335.055857] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2335.065164] LustreError: 100696:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2335.072279] LustreError: 100696:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2335.079389] LustreError: 100696:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2335.160620] Lustre: server umount lustre-OST0000 complete [ 2335.165249] LustreError: 100696:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2344.735279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2344.952453] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2344.958438] Lustre: Found index 0 for lustre-OST0000, updating log [ 2344.970870] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2345.093671] Lustre: lustre-OST0000: new disk, initializing [ 2345.099869] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2345.106743] Lustre: 101235:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2345.350615] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2351.237038] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2355.597516] Lustre: Failing over lustre-OST0000 [ 2355.759435] Lustre: server umount lustre-OST0000 complete [ 2362.852423] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2362.874735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2362.883319] Lustre: Skipped 3 previous similar messages [ 2367.013776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2367.026655] Lustre: Skipped 1 previous similar message [ 2370.150091] Lustre: server umount lustre-MDT0000 complete [ 2372.070815] LustreError: 98144:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2372.099443] LustreError: 98144:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2374.199855] LustreError: 100715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834831 with bad export cookie 1164316001551254917 [ 2374.205135] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2374.213721] LustreError: 100715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2374.673929] Lustre: server umount lustre-MDT0001 complete [ 2382.297234] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 08:47:19 (1787834839) [ 2397.664376] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 2406.332794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2406.588050] LustreError: 103581:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2406.732132] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2410.422290] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2417.925129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2422.724501] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2425.413810] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2430.505158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2431.116814] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2432.164914] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2432.201362] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2432.202833] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2432.217791] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2436.593148] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2443.325513] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2443.387432] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2443.469639] Lustre: lustre-OST0001: new disk, initializing [ 2443.475423] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2443.483266] Lustre: 105899:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2443.782493] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2443.789837] Lustre: Skipped 2 previous similar messages [ 2448.727907] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2452.033944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2452.043979] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2452.118196] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2454.650859] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2460.877546] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2464.740138] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2464.760420] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2464.781013] Lustre: Skipped 1 previous similar message [ 2464.795188] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2470.546820] Lustre: server umount lustre-MDT0001 complete [ 2471.089546] LustreError: 103578:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2471.112248] LustreError: 103578:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2477.498919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2477.570868] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2477.575665] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2477.611476] Lustre: 103571:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2479.604354] LustreError: 107486:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2479.609480] LustreError: 107486:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2479.621979] LustreError: 103573:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2479.727852] LustreError: 107498:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.108@tcp: cfg command failed: rc = -17 [ 2479.737322] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2479.737322] [ 2481.770137] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2483.705433] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2489.192323] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 08:49:06 (1787834946) [ 2503.005540] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 2517.254223] Lustre: 109508:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2537.021168] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2540.204903] Lustre: 110643:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2619.872537] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2619.874904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2619.889013] Lustre: Skipped 5 previous similar messages [ 2619.889611] Lustre: Skipped 4 previous similar messages [ 2624.005414] Lustre: server umount lustre-MDT0000 complete [ 2624.993918] LustreError: 103577:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2625.009154] LustreError: 103577:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 2627.668089] LustreError: 103563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835084 with bad export cookie 1164316001551255925 [ 2627.673615] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2627.684896] LustreError: 103563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2627.958832] Lustre: server umount lustre-MDT0001 complete [ 2642.020246] Lustre: server umount lustre-OST0000 complete [ 2655.818806] Lustre: server umount lustre-OST0001 complete [ 2661.080700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2670.610221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2677.809778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2686.233277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2706.937902] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 2715.359412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2715.373749] Lustre: lustre-MDT0000: reset Object Index mappings [ 2715.755816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2715.761932] Lustre: Skipped 1 previous similar message [ 2719.111898] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2725.536529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2725.553269] Lustre: lustre-MDT0001: reset Object Index mappings [ 2729.713749] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2732.150580] Lustre: 115739:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2737.782350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2740.404527] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:382 to 0x280000401:418) [ 2743.648068] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2750.983250] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:199 to 0x280000400:225) [ 2751.394530] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2757.109623] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 2757.112581] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 2757.641787] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2765.401859] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2768.359098] Lustre: 117616:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2809.622180] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 08:54:26 (1787835266) [ 2838.497840] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2838.504985] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2838.522276] Lustre: Skipped 1 previous similar message [ 2838.529298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2838.536150] Lustre: Skipped 2 previous similar messages [ 2843.563260] Lustre: server umount lustre-MDT0000 complete [ 2844.135981] LustreError: 115702:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2844.164104] LustreError: 115702:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 2846.890761] LustreError: 114575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835303 with bad export cookie 1164316001551303000 [ 2846.907230] LustreError: 114575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2846.910173] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2853.500366] Lustre: server umount lustre-MDT0001 complete [ 2871.263898] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2871.419763] Lustre: server umount lustre-OST0000 complete [ 2880.686585] Lustre: server umount lustre-OST0001 complete [ 2886.201489] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 2893.934923] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 2932.859955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2933.149929] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2933.176933] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2933.247849] Lustre: lustre-MDT0000: new disk, initializing [ 2933.301158] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2933.304885] Lustre: Skipped 3 previous similar messages [ 2933.345703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2937.379726] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2943.627058] Lustre: server umount lustre-MDT0000 complete [ 2954.949134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2955.156664] Lustre: Found index 0 for lustre-MDT0000, updating log [ 2955.172402] Lustre: 121885:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2955.186475] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 2960.534245] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2966.273260] Lustre: server umount lustre-MDT0000 complete [ 2975.369485] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 08:57:11 (1787835431) [ 2977.063216] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 2978.948554] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 08:57:15 (1787835435) [ 2994.966855] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 3003.853611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3008.015766] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3015.252061] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3015.359681] Lustre: 123934:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3015.420997] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3015.429784] Lustre: Skipped 1 previous similar message [ 3015.498331] Lustre: lustre-MDT0001: new disk, initializing [ 3015.612135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3015.620748] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3019.404913] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3023.771162] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3029.432292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3029.671660] Lustre: lustre-OST0000: new disk, initializing [ 3029.675086] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3029.679887] Lustre: 125513:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3031.532524] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3031.545360] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3031.649302] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3034.813400] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3043.052601] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3043.281681] Lustre: lustre-OST0001: new disk, initializing [ 3043.286158] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3043.297655] Lustre: 126382:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3044.469294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3044.481597] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3044.572327] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3048.237066] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3055.940556] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3058.853568] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3416.181982] Lustre: 132799:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3416.188194] Lustre: 132799:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3416.193292] Lustre: 132799:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3416.202895] Lustre: 132799:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3416.207779] Lustre: 132799:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3416.215848] Lustre: 132799:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3647.904966] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 09:08:24 (1787836104) [ 3662.307887] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3662.316228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3662.327269] Lustre: Skipped 8 previous similar messages [ 3662.339774] Lustre: Skipped 11 previous similar messages [ 3664.466266] Lustre: server umount lustre-MDT0000 complete [ 3667.443543] LustreError: 123943:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3667.483267] LustreError: 123943:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 3667.792096] LustreError: 132824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787836124 with bad export cookie 1164316001551368408 [ 3667.809495] LustreError: 132824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3667.816737] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3672.544706] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3672.550474] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3672.562616] Lustre: Skipped 1 previous similar message [ 3672.572123] Lustre: Skipped 1 previous similar message [ 3682.271102] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3682.781762] Lustre: server umount lustre-MDT0001 complete [ 3693.562820] Lustre: server umount lustre-OST0000 complete [ 3703.483372] Lustre: server umount lustre-OST0001 complete [ 3710.458448] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 3721.858977] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 3760.016105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3760.186282] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3760.201645] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3760.261674] Lustre: lustre-MDT0000: new disk, initializing [ 3760.328987] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3760.333658] Lustre: Skipped 5 previous similar messages [ 3760.348726] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3764.105249] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3774.026945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3774.124839] Lustre: 135841:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3774.232900] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3774.238793] Lustre: Skipped 1 previous similar message [ 3774.361795] Lustre: lustre-MDT0001: new disk, initializing [ 3774.505634] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3774.517113] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3779.128662] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3787.798888] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3792.817784] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3798.264399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3798.533572] Lustre: lustre-OST0000: new disk, initializing [ 3798.538232] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3798.546229] Lustre: 137811:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3800.114049] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3800.127993] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3800.229278] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3804.104662] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3811.553207] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3819.159460] Lustre: lustre-MDD0000: changelog on [ 3821.419946] Lustre: lustre-MDD0001: changelog on [ 5423.387494] Lustre: lustre-MDD0001: changelog off [ 5424.850664] Lustre: 135851:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 5425.596105] Lustre: lustre-MDD0000: changelog off [ 5433.315746] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5433.329853] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5433.349677] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5433.358612] Lustre: Skipped 2 previous similar messages [ 5433.827933] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5433.841598] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5436.531085] Lustre: server umount lustre-OST0000 complete [ 5443.565552] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5443.573971] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5443.588339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5443.597417] Lustre: Skipped 1 previous similar message [ 5449.184328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5449.190821] Lustre: Skipped 3 previous similar messages [ 5453.283098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5453.302548] Lustre: Skipped 1 previous similar message [ 5454.303702] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5454.554173] Lustre: server umount lustre-MDT0000 complete [ 5457.805709] LustreError: 135834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787837914 with bad export cookie 1164316001552146374 [ 5457.809896] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5457.817085] LustreError: 135834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 5458.124776] Lustre: server umount lustre-MDT0001 complete [ 5467.217483] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 5469.713894] Key type lgssc unregistered [ 5469.970695] LNet: 146910:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5469.980025] LNetError: 146910:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5469.999770] LNet: Removed LNI 192.168.206.108@tcp [ 5470.722260] Key type .llcrypt unregistered [ 5470.726111] Key type ._llcrypt unregistered [ 5488.347740] Key type ._llcrypt registered [ 5488.350672] Key type .llcrypt registered [ 5488.468831] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 5503.534172] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 5504.562468] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5504.583025] alg: No test for adler32 (adler32-zlib) [ 5505.728761] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 5505.948872] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 5507.583564] Key type lgssc registered [ 5508.643862] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5540.948089] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5540.991776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5542.333420] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5542.351092] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5542.422548] Lustre: lustre-MDT0000: new disk, initializing [ 5542.533829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5542.561284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5547.488914] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5559.163861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5559.340450] Lustre: 150110:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5559.410834] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5559.414360] Lustre: Skipped 1 previous similar message [ 5559.496842] Lustre: lustre-MDT0001: new disk, initializing [ 5559.571360] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5559.641790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5559.653341] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5563.612980] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5571.343218] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5575.827756] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5581.727233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5582.001531] Lustre: lustre-OST0000: new disk, initializing [ 5582.004281] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5582.017184] Lustre: 152080:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5582.118147] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5587.476975] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5592.135043] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5592.155858] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5592.261726] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5595.676943] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5601.663472] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_module obdecho/obdecho [ 5603.622532] LustreError: 153083:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 5603.629761] LustreError: 153083:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 5605.537699] LustreError: 153180:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 5616.096618] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5616.108116] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5616.124263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5618.151736] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5618.154945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5618.171748] Lustre: Skipped 1 previous similar message [ 5618.183839] Lustre: Skipped 2 previous similar messages [ 5622.075947] Lustre: server umount lustre-MDT0000 complete [ 5625.672906] LustreError: 152078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838082 with bad export cookie 5895744770277527825 [ 5625.681529] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5625.689967] LustreError: 152078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5626.021944] Lustre: server umount lustre-MDT0001 complete [ 5640.077692] Lustre: server umount lustre-OST0000 complete [ 5657.673034] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 5667.444442] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 5699.338571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5699.566843] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5699.606237] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5699.735948] Lustre: lustre-MDT0000: new disk, initializing [ 5699.874242] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5699.892175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5704.418313] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5714.607780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5714.771623] Lustre: 156895:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5714.829782] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5714.835299] Lustre: Skipped 1 previous similar message [ 5714.905453] Lustre: lustre-MDT0001: new disk, initializing [ 5715.048788] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5715.092902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5715.103260] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5719.579439] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5730.308174] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5737.239732] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5743.937942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5744.289660] Lustre: lustre-OST0000: new disk, initializing [ 5744.296246] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5744.305926] Lustre: 158864:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5744.405867] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5750.486225] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5752.394332] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5752.408993] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5752.545992] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5759.639415] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5763.314259] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5763.477405] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5767.213670] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5767.438797] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5773.283559] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5773.290515] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5773.312937] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5773.326087] Lustre: Skipped 1 previous similar message [ 5775.059788] Lustre: server umount lustre-OST0000 complete [ 5778.403081] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5778.406401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5778.425554] Lustre: Skipped 2 previous similar messages [ 5778.449889] Lustre: Skipped 1 previous similar message [ 5783.522227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5783.531791] Lustre: Skipped 1 previous similar message [ 5784.753959] Lustre: server umount lustre-MDT0000 complete [ 5788.641652] LustreError: 156901:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5788.692831] LustreError: 156901:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 5789.051218] LustreError: 156886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838245 with bad export cookie 5895744770277528784 [ 5789.053872] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5789.065508] LustreError: 156886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5789.537579] Lustre: server umount lustre-MDT0001 complete [ 5793.492145] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 09:44:09 (1787838249) [ 5795.302406] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 5797.339705] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 09:44:13 (1787838253) [ 5804.699311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5805.309275] LustreError: 161276:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5805.406741] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5809.205438] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5810.657954] LustreError: 161277:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5815.777074] LustreError: 161276:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5817.291503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5822.390557] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5829.801470] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5835.505888] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5841.561480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5841.867341] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5841.871705] Lustre: Skipped 1 previous similar message [ 5847.022257] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5855.139638] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5861.352520] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5861.363354] Lustre: Skipped 1 previous similar message [ 5861.363790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5861.381752] Lustre: Skipped 1 previous similar message [ 5867.317225] Lustre: server umount lustre-MDT0000 complete [ 5870.903747] LustreError: 161257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838327 with bad export cookie 5895744770277529666 [ 5870.917700] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5870.938780] LustreError: 161257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5871.269642] LustreError: 161273:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5871.584692] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5877.365306] Lustre: server umount lustre-MDT0001 complete [ 5884.187830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5895.736057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5895.933431] LustreError: 165116:0:(llog.c:1655:llog_backup()) MGC192.168.206.108@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 5895.947112] Lustre: 165116:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.108@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 5900.261301] LustreError: 165121:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5900.274230] LustreError: 165121:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5917.354776] LustreError: 165121:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5917.383649] LustreError: 165121:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 5936.106842] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 5936.119789] Lustre: Skipped 9 previous similar messages [ 5950.946910] LustreError: 165121:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5950.958104] LustreError: 165121:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 23 previous similar messages [ 5956.575362] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 5956.584271] Lustre: 165116:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.108@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 5997.549435] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6001.388742] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6002.664437] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6007.828393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6015.970368] LustreError: 165123:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6015.974486] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x51d1e4a1aec27178 [ 6015.982938] LustreError: 165123:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 45 previous similar messages [ 6015.991197] Lustre: MGC192.168.206.108@tcp: Connection restored to 0@lo (at 0@lo) [ 6020.062319] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6021.619581] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6026.294690] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6027.516257] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6032.159816] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6033.500798] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6052.320144] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6052.329495] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6052.335807] LustreError: Skipped 1 previous similar message [ 6052.340191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6052.350547] Lustre: Skipped 1 previous similar message [ 6052.371055] Lustre: Skipped 2 previous similar messages [ 6055.426529] Lustre: server umount lustre-MDT0000 complete [ 6058.719777] LustreError: 164058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838515 with bad export cookie 5895744770277536120 [ 6058.727782] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6058.734017] LustreError: 164058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 6059.020284] Lustre: server umount lustre-MDT0001 complete [ 6072.952404] Lustre: server umount lustre-OST0000 complete [ 6092.868705] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 6103.281456] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 6133.626809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6133.865949] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6133.922562] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6134.041560] Lustre: lustre-MDT0000: new disk, initializing [ 6134.147522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6134.155057] Lustre: Skipped 1 previous similar message [ 6134.169157] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6138.886477] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6149.500653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6149.644586] Lustre: 170222:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6149.727358] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6149.989362] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6154.609040] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6164.514668] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6169.945049] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6176.346841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6176.671035] Lustre: lustre-OST0000: new disk, initializing [ 6176.675889] Lustre: Skipped 1 previous similar message [ 6176.684227] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6176.688708] Lustre: Skipped 2 previous similar messages [ 6176.693453] Lustre: 172187:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6182.308415] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6186.053789] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6186.063841] Lustre: Skipped 1 previous similar message [ 6186.070638] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6186.144278] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6190.569631] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6195.022259] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6195.236380] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6199.295782] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6199.496824] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6201.827370] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6201.827814] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6201.846088] Lustre: Skipped 2 previous similar messages [ 6207.403788] Lustre: server umount lustre-OST0000 complete [ 6211.553052] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6217.531968] Lustre: server umount lustre-MDT0000 complete [ 6220.972779] LustreError: 170213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838677 with bad export cookie 5895744770277552955 [ 6220.977576] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6220.990253] LustreError: 170213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6221.522477] Lustre: server umount lustre-MDT0001 complete [ 6229.455802] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 09:51:26 (1787838686) [ 6234.749642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6235.245030] LustreError: 174597:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6235.278646] LustreError: 174597:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6239.553354] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6249.071850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6254.907585] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6263.604896] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6269.704560] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6276.707701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6285.308720] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6293.452642] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6305.970189] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 09:52:42 (1787838762) [ 6310.370342] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6310.373835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6310.408872] Lustre: Skipped 3 previous similar messages [ 6310.416511] Lustre: Skipped 9 previous similar messages [ 6315.511328] Lustre: server umount lustre-OST0000 complete [ 6325.702452] Lustre: server umount lustre-MDT0000 complete [ 6329.008177] LustreError: 174579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838785 with bad export cookie 5895744770277553844 [ 6329.014641] LustreError: 174579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6329.022888] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6329.347246] Lustre: server umount lustre-MDT0001 complete [ 6340.564808] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 6343.582718] Key type lgssc unregistered [ 6343.857794] LNet: 178328:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6343.864092] LNetError: 178328:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6343.887340] LNet: Removed LNI 192.168.206.108@tcp [ 6344.622291] Key type .llcrypt unregistered [ 6344.624523] Key type ._llcrypt unregistered [ 6363.233900] Key type ._llcrypt registered [ 6363.236637] Key type .llcrypt registered [ 6363.379209] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 6364.220939] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6364.251315] alg: No test for adler32 (adler32-zlib) [ 6365.229370] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 6365.448221] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 6367.135120] Key type lgssc registered [ 6368.281536] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6376.075654] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6376.123851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6391.775372] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6396.895468] LustreError: 179574:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.108@tcp: failed processing log, type 4: rc = -110 [ 6402.015621] LustreError: 179574:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6402.026382] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 6402.034623] LustreError: MGC192.168.206.108@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 6402.047324] LustreError: 179574:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 6402.064534] LustreError: 179574:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -22 [ 6402.074996] LustreError: 179574:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 6402.144872] Lustre: server umount lustre-OST0000 complete [ 6402.146754] LustreError: 179574:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6410.593690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6412.069275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6416.529497] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6421.472915] LustreError: 180074:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6421.498746] LustreError: 180074:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6425.482733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6426.021371] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6430.707757] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6437.702970] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6443.369933] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6449.613658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6449.783043] LustreError: 181890:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6449.789704] Lustre: 181890:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.206.108@tcp: invalid local config log lustre-OST0000: rc = -22 [ 6449.898256] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6455.071021] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6462.459270] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6467.060273] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6467.073103] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6467.099146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6472.672242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6472.682671] Lustre: Skipped 4 previous similar messages [ 6477.792646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6477.806022] Lustre: Skipped 2 previous similar messages [ 6479.327106] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6479.497057] Lustre: server umount lustre-MDT0000 complete [ 6482.270204] LustreError: 180056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838939 with bad export cookie 11772708240152230704 [ 6482.278886] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6482.288147] LustreError: 180056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6482.554540] Lustre: server umount lustre-MDT0001 complete [ 6496.653869] Lustre: server umount lustre-OST0000 complete [ 6505.577689] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 6514.788620] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 6551.471128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6551.705556] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6551.725218] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6551.815700] Lustre: lustre-MDT0000: new disk, initializing [ 6551.887218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6551.897371] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6556.329670] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6566.818901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6566.945920] Lustre: 185417:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6566.991566] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6566.997401] Lustre: Skipped 1 previous similar message [ 6567.062360] Lustre: lustre-MDT0001: new disk, initializing [ 6567.170121] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6567.246971] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6567.270403] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6571.969181] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6581.284440] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6586.655614] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6593.636380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6593.880298] Lustre: lustre-OST0000: new disk, initializing [ 6593.888366] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6593.898253] Lustre: 187385:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6593.989788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6599.466786] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6600.307705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6600.314652] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6600.383311] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6607.530089] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6611.834858] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6612.033716] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6616.429647] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6616.614772] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6621.152763] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6621.157501] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6621.165892] LustreError: Skipped 1 previous similar message [ 6621.168440] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6621.205797] Lustre: Skipped 3 previous similar messages [ 6624.463065] Lustre: server umount lustre-OST0000 complete [ 6628.835246] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6628.848718] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6628.861704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6628.870700] Lustre: Skipped 1 previous similar message [ 6631.399404] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6634.387648] Lustre: server umount lustre-MDT0000 complete [ 6636.515098] LustreError: 185430:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6636.531144] LustreError: 185430:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6637.694055] LustreError: 185410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787839094 with bad export cookie 11772708240152231418 [ 6637.695111] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6637.700699] LustreError: 185410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6637.950294] Lustre: server umount lustre-MDT0001 complete [ 6645.094619] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 09:58:21 (1787839101) [ 6659.035024] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 6661.540956] Key type lgssc unregistered [ 6661.759497] LNet: 190333:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6661.763857] LNetError: 190333:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6661.793251] LNet: Removed LNI 192.168.206.108@tcp [ 6662.742481] Key type .llcrypt unregistered [ 6662.747228] Key type ._llcrypt unregistered [ 6679.079583] Key type ._llcrypt registered [ 6679.081213] Key type .llcrypt registered [ 6679.179176] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 6680.114317] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6680.130085] alg: No test for adler32 (adler32-zlib) [ 6681.181933] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 6681.513724] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 6683.183198] Key type lgssc registered [ 6684.617889] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6691.387692] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6691.412547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6692.908677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6697.204001] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6703.072575] LustreError: 191556:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6703.089079] LustreError: 191556:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6704.399130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6704.895891] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6709.000333] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6714.575301] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6718.707694] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6724.536404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6724.714352] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6724.718578] LustreError: 193417:0:(llog.c:1696:llog_backup()) MGC192.168.206.108@tcp: failed to backup log lustre-OST0000: rc = -28 [ 6724.731088] Lustre: 193417:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.206.108@tcp: can't backup local config lustre-OST0000: rc = -28 [ 6724.825886] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6729.952414] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6736.074507] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6740.964920] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6740.967878] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6740.990734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6742.497660] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6742.499292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6742.516380] Lustre: Skipped 1 previous similar message [ 6747.619803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6747.628247] Lustre: Skipped 2 previous similar messages [ 6752.225310] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6752.472940] Lustre: server umount lustre-MDT0000 complete [ 6752.740246] LustreError: 191555:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6752.755697] LustreError: 191555:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6755.516641] LustreError: 191536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787839212 with bad export cookie 9530033905521688111 [ 6755.527577] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6755.530697] LustreError: 191536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6755.768294] Lustre: server umount lustre-MDT0001 complete [ 6769.108866] Lustre: server umount lustre-OST0000 complete [ 6778.516845] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 6788.944531] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 6817.894318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6818.044203] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6818.070630] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6818.139541] Lustre: lustre-MDT0000: new disk, initializing [ 6818.206259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6818.221242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6821.569359] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6830.349438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6830.462373] Lustre: 196955:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6830.525802] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6830.529448] Lustre: Skipped 1 previous similar message [ 6830.533490] LustreError: 197736:0:(llog.c:1696:llog_backup()) MGC192.168.206.108@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 6830.540217] LustreError: 197736:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 6830.545940] Lustre: 197736:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.108@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 6830.563823] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6830.569203] Lustre: Skipped 1 previous similar message [ 6830.639215] Lustre: lustre-MDT0001: new disk, initializing [ 6830.727086] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6830.754569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6830.771076] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6834.304740] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6841.270035] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6845.972877] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6852.238672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6852.457012] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6852.458575] Lustre: Skipped 1 previous similar message [ 6852.460296] LustreError: 198896:0:(llog.c:1696:llog_backup()) MGC192.168.206.108@tcp: failed to backup log lustre-OST0000: rc = -28 [ 6852.463954] LustreError: 198896:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 6852.467339] Lustre: 198896:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.108@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 6852.473577] Lustre: 198896:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 6852.487340] Lustre: lustre-OST0000: new disk, initializing [ 6852.490183] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6852.494116] Lustre: 198920:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6852.580341] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6858.605805] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6859.371580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6859.379953] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6859.549850] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6868.401383] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6872.801033] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6873.013536] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6877.176615] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6877.446580] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6880.227102] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6880.230264] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6880.240213] LustreError: Skipped 1 previous similar message [ 6880.256077] Lustre: Skipped 1 previous similar message [ 6880.259208] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6885.350520] LustreError: 198905:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6885.364797] LustreError: 198905:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6885.496855] Lustre: server umount lustre-OST0000 complete [ 6890.470480] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6890.471740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6890.490782] Lustre: Skipped 1 previous similar message [ 6890.499952] Lustre: Skipped 2 previous similar messages [ 6894.728686] Lustre: server umount lustre-MDT0000 complete [ 6895.585150] LustreError: 196966:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6898.001946] LustreError: 196947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787839354 with bad export cookie 9530033905521688832 [ 6898.008288] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6898.024987] LustreError: 196947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6898.301702] Lustre: server umount lustre-MDT0001 complete [ 6905.788907] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 10:02:42 (1787839362) [ 6922.054616] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 6932.144890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6932.692760] LustreError: 202048:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6932.712415] LustreError: 202048:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6932.745935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6937.455811] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6942.182713] LustreError: 202049:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6942.205585] LustreError: 202049:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6945.539185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6950.122389] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6952.834136] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6958.823065] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6964.492281] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6971.962227] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6972.123783] Lustre: lustre-OST0001: new disk, initializing [ 6972.127434] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6972.132260] Lustre: 204359:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6972.220073] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6972.224278] Lustre: Skipped 2 previous similar messages [ 6977.696671] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6980.237879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6980.251762] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6980.386784] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6987.356539] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6996.799777] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6998.307157] Lustre: DEBUG MARKER: ADD OST3 [ 7002.157121] loop: module loaded [ 7002.225577] loop0: detected capacity change from 0 to 204800000 [ 7002.392895] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7002.402212] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7002.410396] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7006.853787] loop0: detected capacity change from 0 to 204800000 [ 7009.125394] Lustre: DEBUG MARKER: STOP OST3 [ 7011.642191] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7011.807782] Lustre: lustre-OST0003: new disk, initializing [ 7011.813344] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7011.825466] Lustre: 206293:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7013.701067] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7013.958704] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7013.972709] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7014.061633] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7014.192937] Lustre: Failing over lustre-OST0003 [ 7014.764217] Lustre: server umount lustre-OST0003 complete [ 7015.776402] LustreError: 206447:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7018.715455] LustreError: 203536:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.206.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7018.986951] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7018.991911] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7019.021902] Lustre: Skipped 1 previous similar message [ 7019.585819] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7030.979527] Lustre: DEBUG MARKER: START OST3 again [ 7036.688844] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7037.099413] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7037.104244] Lustre: Skipped 1 previous similar message [ 7037.120922] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7038.246022] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7038.745059] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7038.748820] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 7042.274875] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7061.516233] Lustre: Failing over lustre-OST0003 [ 7061.584192] Lustre: server umount lustre-OST0003 complete [ 7063.013218] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7063.028741] Lustre: Skipped 1 previous similar message [ 7063.028766] LustreError: 203536:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7063.050934] LustreError: 203536:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 7078.371708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7084.818468] Lustre: server umount lustre-MDT0000 complete [ 7088.858674] LustreError: 202028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787839545 with bad export cookie 9530033905521689714 [ 7088.864691] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7088.870445] LustreError: 202028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7093.733707] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7093.744663] Lustre: Skipped 4 previous similar messages [ 7095.350266] Lustre: server umount lustre-MDT0001 complete [ 7098.787979] Lustre: server umount lustre-OST0000 complete [ 7102.142711] Lustre: server umount lustre-OST0001 complete [ 7108.782915] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 7118.065538] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 7152.932310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7153.086978] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7153.103834] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7153.201317] Lustre: lustre-MDT0000: new disk, initializing [ 7153.336735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7157.532578] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7166.544748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7166.653901] Lustre: 211822:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7166.850053] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7166.859461] Lustre: Skipped 1 previous similar message [ 7166.891546] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7171.326580] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7179.725929] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7184.983652] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7190.709076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7190.952837] Lustre: lustre-OST0000: new disk, initializing [ 7190.959617] Lustre: Skipped 1 previous similar message [ 7190.966818] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7190.970951] Lustre: Skipped 2 previous similar messages [ 7190.981253] Lustre: 213790:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7192.259486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7192.268058] Lustre: Skipped 1 previous similar message [ 7192.280457] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7192.375173] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7196.932799] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7205.310892] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7209.117340] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7209.290264] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7212.858830] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7213.022170] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7218.144363] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7218.151955] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7218.161555] LustreError: Skipped 1 previous similar message [ 7218.174188] Lustre: Skipped 1 previous similar message [ 7218.180513] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7218.191100] Lustre: Skipped 10 previous similar messages [ 7220.608381] Lustre: server umount lustre-OST0000 complete [ 7228.384719] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7230.865809] Lustre: server umount lustre-MDT0000 complete [ 7233.512918] LustreError: 211827:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7233.539093] LustreError: 211827:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 7234.459766] LustreError: 212617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787839691 with bad export cookie 9530033905521692010 [ 7234.467585] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7234.468041] LustreError: 212617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7234.672729] Lustre: server umount lustre-MDT0001 complete [ 7238.115951] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 10:08:14 (1787839694) [ 7251.736188] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 7261.233294] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 7289.726286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7289.884328] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7289.902637] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7289.970232] Lustre: lustre-MDT0000: new disk, initializing [ 7290.058570] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7293.089554] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7301.612355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7301.724444] Lustre: 218396:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7301.933724] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7305.046930] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7312.812270] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7317.114559] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7322.416918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7322.627342] Lustre: 220360:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7324.264500] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7327.737741] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7336.461605] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7340.820932] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7341.012769] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7345.356573] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7345.581538] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7349.734309] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7349.736275] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7349.737443] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7349.737450] Lustre: Skipped 2 previous similar messages [ 7349.749535] LustreError: Skipped 1 previous similar message [ 7349.779875] Lustre: Skipped 4 previous similar messages [ 7353.601255] Lustre: server umount lustre-OST0000 complete [ 7363.665765] Lustre: server umount lustre-MDT0000 complete [ 7365.094832] LustreError: 219195:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7366.853467] LustreError: 220359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787839823 with bad export cookie 9530033905521692892 [ 7366.861901] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7366.871431] LustreError: 220359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7367.133909] Lustre: server umount lustre-MDT0001 complete [ 7374.360481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7379.419987] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7387.932747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7393.522930] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7400.026457] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7405.818831] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7412.433110] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7419.150450] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7426.430162] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7439.341478] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7439.362429] LustreError: Skipped 1 previous similar message [ 7439.379446] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7439.389589] Lustre: Skipped 4 previous similar messages [ 7442.517401] Lustre: server umount lustre-OST0000 complete [ 7452.613482] Lustre: server umount lustre-MDT0000 complete [ 7455.642988] LustreError: 222557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787839912 with bad export cookie 9530033905521693767 [ 7455.658233] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7455.662930] LustreError: 222557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7456.075483] Lustre: server umount lustre-MDT0001 complete [ 7562.214197] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 7564.468594] Key type lgssc unregistered [ 7564.723269] LNet: 226246:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7564.728890] LNetError: 226246:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7564.747186] LNet: Removed LNI 192.168.206.108@tcp [ 7565.539253] Key type .llcrypt unregistered [ 7565.541334] Key type ._llcrypt unregistered [ 7587.117226] Key type ._llcrypt registered [ 7587.120363] Key type .llcrypt registered [ 7587.216498] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 7603.808506] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 7604.692993] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7604.766437] alg: No test for adler32 (adler32-zlib) [ 7605.826644] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 7606.130830] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 7607.791454] Key type lgssc registered [ 7608.997101] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7637.221653] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7637.238916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7638.472084] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7638.498451] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7638.565441] Lustre: lustre-MDT0000: new disk, initializing [ 7638.636905] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7638.650645] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7641.862304] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7649.840666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7649.946798] Lustre: 229595:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7650.005397] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7650.008724] Lustre: Skipped 1 previous similar message [ 7650.073708] Lustre: lustre-MDT0001: new disk, initializing [ 7650.160858] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7650.196482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7650.202208] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7653.899655] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7662.030249] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7666.581506] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7671.692584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7671.913604] Lustre: lustre-OST0000: new disk, initializing [ 7671.918602] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7671.925117] Lustre: 231564:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7672.012640] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7676.892364] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7677.509925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7677.524502] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7677.646940] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7684.955742] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7688.937370] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7689.153289] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7693.547451] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7693.756444] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7698.407182] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7698.408633] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7698.410600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7698.415665] Lustre: Skipped 1 previous similar message [ 7698.425470] LustreError: Skipped 1 previous similar message [ 7701.584754] Lustre: server umount lustre-OST0000 complete [ 7706.593088] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7706.598501] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7706.619174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7706.628584] Lustre: Skipped 1 previous similar message [ 7708.641223] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7708.644185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7708.667506] Lustre: Skipped 1 previous similar message [ 7713.774944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7713.780887] Lustre: Skipped 1 previous similar message [ 7718.879743] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7718.882607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7718.899034] Lustre: Skipped 1 previous similar message [ 7719.073351] Lustre: server umount lustre-MDT0000 complete [ 7722.660804] LustreError: 230401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787840179 with bad export cookie 2507680712513852935 [ 7722.667871] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7722.671330] LustreError: 230401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7722.962234] Lustre: server umount lustre-MDT0001 complete [ 7741.804198] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 7753.231346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7753.539983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7757.368475] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7758.820172] LustreError: 234501:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7758.831485] LustreError: 234501:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7763.954652] LustreError: 234500:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7765.530531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7765.921106] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7770.419254] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7772.903383] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7778.760992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7779.136042] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7784.758701] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7791.948830] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7792.141792] Lustre: lustre-OST0001: new disk, initializing [ 7792.145637] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7792.151825] Lustre: 236807:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7792.406068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7792.417938] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7792.538436] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7798.053387] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7808.424521] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7812.518423] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7823.162466] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 10:18:00 (1787840280) [ 7827.433901] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7827.436921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7827.444268] Lustre: Skipped 3 previous similar messages [ 7827.444962] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7827.468897] Lustre: Skipped 3 previous similar messages [ 7839.203917] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7839.323859] Lustre: server umount lustre-MDT0000 complete [ 7842.805042] LustreError: 234501:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7842.841113] LustreError: 234501:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7843.018821] LustreError: 234481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787840299 with bad export cookie 2507680712513853817 [ 7843.032182] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7843.053820] LustreError: 234481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7843.570716] Lustre: server umount lustre-MDT0001 complete [ 7857.466943] Lustre: server umount lustre-OST0000 complete [ 7872.109747] Lustre: server umount lustre-OST0001 complete [ 7877.575574] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 7886.564731] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 7920.931315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7921.193971] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7921.240318] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7921.347217] Lustre: lustre-MDT0000: new disk, initializing [ 7921.507174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7921.512044] Lustre: Skipped 1 previous similar message [ 7921.568635] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7926.168555] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7936.241922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7936.320618] Lustre: 240751:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7936.382847] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7936.386135] Lustre: Skipped 1 previous similar message [ 7936.503911] Lustre: lustre-MDT0001: new disk, initializing [ 7936.650488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7936.657773] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7941.272930] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7950.634473] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7955.695223] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7961.738626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7961.925256] Lustre: lustre-OST0000: new disk, initializing [ 7961.928306] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7961.932638] Lustre: 242720:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7961.994834] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7961.997347] Lustre: Skipped 1 previous similar message [ 7963.970347] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7963.978674] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7964.125740] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7967.283458] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7975.178937] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7978.426993] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7978.584296] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7981.777549] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7981.934478] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7984.608980] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7984.614236] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7984.625055] LustreError: Skipped 1 previous similar message [ 7984.626748] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7984.626754] Lustre: Skipped 8 previous similar messages [ 7984.658773] Lustre: Skipped 1 previous similar message [ 7989.432829] Lustre: server umount lustre-OST0000 complete [ 7992.800844] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7992.809748] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7998.739845] Lustre: server umount lustre-MDT0000 complete [ 7999.969312] LustreError: 241638:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7999.994340] LustreError: 241638:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8001.868231] LustreError: 240744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787840458 with bad export cookie 2507680712513855959 [ 8001.871518] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8001.880819] LustreError: 240744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8002.304250] Lustre: server umount lustre-MDT0001 complete [ 8008.869626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8009.247491] LustreError: 244934:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8013.114861] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8019.398788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8019.429715] LustreError: 244934:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8019.460292] LustreError: 244934:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8022.822063] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8028.581370] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8033.620495] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8039.710286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8040.073750] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8040.078820] Lustre: Skipped 2 previous similar messages [ 8046.750787] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8053.960213] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8198.117585] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8198.117614] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8198.124916] LustreError: Skipped 1 previous similar message [ 8198.139543] Lustre: Skipped 1 previous similar message [ 8198.142974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8198.150059] Lustre: Skipped 4 previous similar messages [ 8203.567050] Lustre: server umount lustre-OST0000 complete [ 8213.465177] Lustre: server umount lustre-MDT0000 complete [ 8213.478184] LustreError: 244935:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8213.503272] LustreError: 244935:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8216.113124] LustreError: 244917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787840673 with bad export cookie 2507680712513856841 [ 8216.119191] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8216.359169] Lustre: server umount lustre-MDT0001 complete [ 8319.908055] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 8322.138943] Key type lgssc unregistered [ 8322.403914] LNet: 248547:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8322.412268] LNetError: 248547:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8322.447134] LNet: Removed LNI 192.168.206.108@tcp [ 8323.369280] Key type .llcrypt unregistered [ 8323.377168] Key type ._llcrypt unregistered [ 8347.112962] Key type ._llcrypt registered [ 8347.116200] Key type .llcrypt registered [ 8347.219272] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 8361.036144] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 8362.004649] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8362.088840] alg: No test for adler32 (adler32-zlib) [ 8363.080490] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 8363.280205] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 8364.935178] Key type lgssc registered [ 8365.746108] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8402.338941] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 8412.654415] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8412.670903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8413.903683] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8413.946205] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8414.004179] Lustre: lustre-MDT0000: new disk, initializing [ 8414.071578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8414.084220] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8418.564414] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8426.862754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8426.955484] Lustre: 252615:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8427.021852] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8427.026705] Lustre: Skipped 1 previous similar message [ 8427.083664] Lustre: lustre-MDT0001: new disk, initializing [ 8427.180210] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8427.220807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8427.233297] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8430.385207] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8434.091488] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8439.653862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8439.864521] Lustre: lustre-OST0000: new disk, initializing [ 8439.867242] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8439.874284] Lustre: 254249:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8439.950971] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8440.670786] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8440.685791] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8440.782516] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8445.876867] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8455.340213] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8455.522663] Lustre: lustre-OST0001: new disk, initializing [ 8455.526866] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8455.538552] Lustre: 255118:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8455.646822] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8460.980188] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8465.473238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8465.479722] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8465.510383] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8470.629944] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8475.680518] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8485.867274] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8485.878189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8485.894249] Lustre: Skipped 3 previous similar messages [ 8485.921480] Lustre: Skipped 3 previous similar messages [ 8490.484927] Lustre: server umount lustre-MDT0000 complete [ 8493.146780] LustreError: 252607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787840950 with bad export cookie 1413670587108151002 [ 8493.150438] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8493.163100] LustreError: 252607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8493.362699] Lustre: server umount lustre-MDT0001 complete [ 8506.968522] Lustre: server umount lustre-OST0000 complete [ 8521.013868] Lustre: server umount lustre-OST0001 complete [ 8535.824325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8544.872582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8551.634360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 8559.147875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 8575.254804] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 8582.949349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8582.969580] Lustre: lustre-MDT0000: reset Object Index mappings [ 8583.343944] LustreError: 260126:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8583.359055] LustreError: 260126:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 8583.406076] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8586.735663] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8588.771829] LustreError: 260127:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8593.892989] LustreError: 260126:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8594.685931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8594.740941] Lustre: lustre-MDT0001: reset Object Index mappings [ 8595.302609] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8599.152460] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8601.391948] Lustre: 261267:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8606.642108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8609.006491] LustreError: 261622:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8609.027089] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8612.153896] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8614.370474] LustreError: 261622:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8619.320624] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8619.531317] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8619.539465] Lustre: Skipped 1 previous similar message [ 8620.534106] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 8624.869375] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8632.011648] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8636.005021] Lustre: 263137:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8640.994822] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8641.002630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8641.007617] Lustre: Skipped 3 previous similar messages [ 8641.014052] Lustre: Skipped 2 previous similar messages [ 8643.364759] Lustre: server umount lustre-MDT0000 complete [ 8646.113964] LustreError: 260127:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8646.140743] LustreError: 260127:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8647.548442] LustreError: 260107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841104 with bad export cookie 1413670587108157799 [ 8647.559033] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8647.891896] Lustre: server umount lustre-MDT0001 complete [ 8661.161607] Lustre: server umount lustre-OST0000 complete [ 8674.904346] Lustre: server umount lustre-OST0001 complete [ 8693.093675] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 10:32:29 (1787841149) [ 8708.110824] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 8717.194138] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 8751.520837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8751.768713] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8751.792834] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8751.848113] Lustre: lustre-MDT0000: new disk, initializing [ 8751.896656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8751.912732] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8756.625749] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8765.825602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8765.969908] Lustre: 267391:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8766.066400] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8766.070913] Lustre: Skipped 1 previous similar message [ 8766.135706] Lustre: lustre-MDT0001: new disk, initializing [ 8766.305515] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8766.320901] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8770.116621] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8779.347737] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8784.848135] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8790.969803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8791.330380] Lustre: lustre-OST0000: new disk, initializing [ 8791.333350] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8791.338372] Lustre: 269362:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8793.252484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8793.275443] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8793.468790] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8797.821647] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8806.950164] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8812.795305] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8813.135186] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8817.341651] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8817.556262] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8824.289017] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8824.296383] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8824.301099] LustreError: Skipped 1 previous similar message [ 8824.316307] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8824.320703] Lustre: Skipped 1 previous similar message [ 8825.531862] Lustre: server umount lustre-OST0000 complete [ 8829.410738] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8829.419659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8829.436829] Lustre: Skipped 2 previous similar messages [ 8829.452934] Lustre: Skipped 2 previous similar messages [ 8834.528257] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8834.539416] Lustre: Skipped 1 previous similar message [ 8843.231128] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8843.585761] Lustre: server umount lustre-MDT0000 complete [ 8844.769553] LustreError: 267399:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8844.797568] LustreError: 267399:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8847.165421] LustreError: 268196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841304 with bad export cookie 1413670587108159892 [ 8847.178388] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8847.179250] LustreError: 268196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 8847.486110] Lustre: server umount lustre-MDT0001 complete [ 8866.945581] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 8877.044526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8877.375144] LustreError: 272295:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8877.419087] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8877.430324] Lustre: Skipped 2 previous similar messages [ 8882.973938] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8891.150902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8895.766306] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8898.782870] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8904.684919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8910.042126] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8916.877426] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8917.115969] Lustre: lustre-OST0001: new disk, initializing [ 8917.119935] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8917.125236] Lustre: 274606:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8918.830521] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8918.839456] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8918.936168] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8922.619271] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8930.587962] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8933.803935] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8938.447754] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 8938.476133] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 8947.681056] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8947.694502] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8947.706556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8947.708942] Lustre: Skipped 2 previous similar messages [ 8950.131319] Lustre: server umount lustre-MDT0000 complete [ 8953.387728] LustreError: 274605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841410 with bad export cookie 1413670587108160774 [ 8953.389836] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8953.399631] LustreError: 274605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8953.744175] Lustre: server umount lustre-MDT0001 complete [ 8967.890540] Lustre: server umount lustre-OST0000 complete [ 8969.184253] Lustre: 250125:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787841410/real 1787841410] req@ffff9fa1cb29d880 x1874686975896960/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787841426 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8969.215812] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8969.235110] Lustre: Skipped 3 previous similar messages [ 8971.120037] Lustre: server umount lustre-OST0001 complete [ 8985.648199] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 8995.073534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8995.439340] LustreError: 277661:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8995.461945] LustreError: 277661:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8999.591313] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9007.325417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9007.763849] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9007.774893] Lustre: Skipped 4 previous similar messages [ 9012.089290] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9014.656250] Lustre: 278801:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9020.404522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9026.280794] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9033.680115] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9039.902441] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9040.383253] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 9040.394589] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 9047.563407] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9050.932705] Lustre: 280678:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9055.159462] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 9055.236663] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 9063.454867] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 10:38:40 (1787841520) [ 9079.267454] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9079.278053] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9079.310152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9079.331489] Lustre: Skipped 4 previous similar messages [ 9084.680823] Lustre: server umount lustre-MDT0000 complete [ 9088.279066] LustreError: 277641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841545 with bad export cookie 1413670587108167333 [ 9088.286505] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9094.768653] Lustre: server umount lustre-MDT0001 complete [ 9104.740785] Lustre: server umount lustre-OST0000 complete [ 9114.692295] Lustre: server umount lustre-OST0001 complete [ 9121.142571] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 9130.649518] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 9174.579247] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 9184.008499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9184.270676] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9184.329445] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9184.393419] Lustre: lustre-MDT0000: new disk, initializing [ 9184.461128] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9189.300400] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9198.268154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9198.391958] Lustre: 284917:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9198.621563] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9202.967799] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9213.491833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9213.760599] Lustre: 286550:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9215.315335] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9219.910654] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9229.239291] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9229.407930] Lustre: 287421:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9231.045360] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9231.053023] Lustre: Skipped 1 previous similar message [ 9231.175521] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9235.360677] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9243.733396] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9246.867422] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9246.870785] Lustre: Skipped 1 previous similar message [ 9259.073556] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 10:41:55 (1787841715) [ 9262.053492] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9262.065501] Lustre: Skipped 6 previous similar messages [ 9262.067816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9262.076591] Lustre: Skipped 11 previous similar messages [ 9267.190731] LustreError: 284923:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9267.198595] Lustre: server umount lustre-MDT0000 complete [ 9267.214378] LustreError: 284923:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 9270.710651] LustreError: 284909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841727 with bad export cookie 1413670587108174963 [ 9270.719736] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9270.731198] LustreError: 284909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9271.054968] Lustre: server umount lustre-MDT0001 complete [ 9284.894453] Lustre: server umount lustre-OST0000 complete [ 9288.169986] Lustre: 250125:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787841729/real 1787841729] req@ffff9fa2ffcd4a80 x1874686976038272/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787841745 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9288.845601] Lustre: server umount lustre-OST0001 complete [ 9297.586325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9298.256660] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9298.264474] Lustre: Skipped 6 previous similar messages [ 9302.557472] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9310.954494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9315.588807] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9322.977868] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9328.018525] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9333.930885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9340.388971] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9347.125542] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9361.894821] loop1: detected capacity change from 0 to 204800000 [ 9362.125776] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9362.136867] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9362.145265] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9367.220719] loop1: detected capacity change from 0 to 204800000 [ 9372.716521] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9372.810262] LustreError: 289654:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs17314: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 9372.819257] LustreError: 289654:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs17314 [ 9372.824190] LustreError: 289654:0:(mgs_handler.c:517:mgs_target_reg()) fs17314-MDT0001: Error checking target index: rc = -13 [ 9372.831815] LustreError: fs17314-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 9372.838212] LustreError: 293383:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [ 9372.848382] LustreError: 293383:0:(tgt_mount.c:2132:server_put_super()) no obd fs17314-MDT0001 [ 9372.859765] LustreError: 293383:0:(tgt_mount.c:116:server_deregister_mount()) fs17314-MDT0001 not registered [ 9372.916474] Lustre: server umount fs17314-MDT0001 complete [ 9372.921731] LustreError: 293383:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 9380.690214] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9380.866768] Lustre: Setting parameter fs17314-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs17314-MDT0001 [ 9380.923534] Lustre: srv-fs17314-MDT0001: No data found on store. Initialize space. [ 9380.927672] Lustre: Skipped 4 previous similar messages [ 9381.031755] Lustre: fs17314-MDT0001: new disk, initializing [ 9381.033956] Lustre: Skipped 3 previous similar messages [ 9381.219903] Lustre: srv-fs17314-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 9381.237621] Lustre: 293850:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs17314-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9381.249058] LustreError: 293850:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs17314-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9385.486882] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9392.040570] Lustre: server umount fs17314-MDT0001 complete [ 9405.374245] loop1: detected capacity change from 0 to 204800000 [ 9410.816678] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9411.112516] Lustre: 295416:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs17314-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9411.119599] LustreError: 295416:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs17314-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9415.059483] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9420.275072] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [ 9420.625515] Lustre: server umount fs17314-MDT0001 complete [ 9425.386795] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [ 9429.691628] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.8@tcp (stopping) [ 9430.498212] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9430.511954] Lustre: Skipped 4 previous similar messages [ 9435.800781] Lustre: server umount lustre-MDT0000 complete [ 9442.192382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9442.307079] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9446.239620] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9447.927753] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9452.390732] loop1: detected capacity change from 0 to 204800000 [ 9457.572256] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9457.863188] Lustre: 297861:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs17314-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9457.871219] LustreError: 297861:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs17314-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9461.181538] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9464.760150] Lustre: server umount fs17314-MDT0001 complete [ 9483.755220] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9487.956691] Lustre: server umount lustre-OST0000 complete [ 9497.631346] Lustre: server umount lustre-MDT0000 complete [ 9500.497863] LustreError: 295907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841957 with bad export cookie 1413670587108177644 [ 9500.502768] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9500.508280] LustreError: 295907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9500.956043] Lustre: server umount lustre-MDT0001 complete [ 9511.215188] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 9513.818573] Key type lgssc unregistered [ 9514.119071] LNet: 300206:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9514.128460] LNetError: 300206:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9514.145117] LNet: Removed LNI 192.168.206.108@tcp [ 9515.086151] Key type .llcrypt unregistered [ 9515.089092] Key type ._llcrypt unregistered [ 9534.708675] Key type ._llcrypt registered [ 9534.714226] Key type .llcrypt registered [ 9534.856605] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 9536.088702] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9536.217182] alg: No test for adler32 (adler32-zlib) [ 9537.321752] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 9537.589057] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 9539.263158] Key type lgssc registered [ 9540.804435] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9548.991322] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9549.022401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9550.407346] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9554.822873] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9560.547876] LustreError: 301480:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9560.556640] LustreError: 301480:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9563.228951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9563.777039] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9568.356404] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9575.278452] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9580.106936] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9586.519367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9586.939194] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9588.969748] LustreError: 303308:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9592.845571] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9594.369165] LustreError: 303309:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9599.863125] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9600.492478] LustreError: 303308:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9600.513833] LustreError: 303308:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9605.610678] LustreError: 303309:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9605.637811] LustreError: 303309:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9615.848878] LustreError: 304338:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9615.870150] LustreError: 304338:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9616.568758] loop1: detected capacity change from 0 to 204800000 [ 9616.732371] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9616.741509] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9616.750988] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9621.464874] loop1: detected capacity change from 0 to 204800000 [ 9626.654292] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9626.778738] Lustre: fs809-OST0000: new disk, initializing [ 9626.781919] Lustre: srv-fs809-OST0000: No data found on store. Initialize space. [ 9626.788442] Lustre: 305214:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9626.856889] Lustre: fs809-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9632.050257] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9633.450658] LustreError: 304338:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9633.485180] LustreError: 304338:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 9640.231993] Lustre: server umount fs809-OST0000 complete [ 9651.324751] loop1: detected capacity change from 0 to 204800000 [ 9656.644529] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9656.828539] Lustre: fs809-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9662.646185] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9667.044280] LustreError: 303309:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9667.066965] LustreError: 303309:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 18 previous similar messages [ 9670.498581] loop2: detected capacity change from 0 to 204800000 [ 9670.741976] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9670.756744] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9670.768031] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9672.184526] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [ 9675.482990] loop2: detected capacity change from 0 to 204800000 [ 9677.288356] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [ 9680.324344] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9680.391843] LustreError: fs809-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 9680.405768] LustreError: fs809-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 9680.415294] LustreError: 308253:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [ 9680.421209] LustreError: 308253:0:(tgt_mount.c:2132:server_put_super()) no obd fs809-OST0001 [ 9680.425309] LustreError: 308253:0:(tgt_mount.c:116:server_deregister_mount()) fs809-OST0001 not registered [ 9680.457878] Lustre: server umount fs809-OST0001 complete [ 9680.462516] LustreError: 308253:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 9688.786458] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9688.977873] Lustre: fs809-OST0001: new disk, initializing [ 9688.984348] Lustre: srv-fs809-OST0001: No data found on store. Initialize space. [ 9688.994928] Lustre: 308805:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9689.079981] Lustre: fs809-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9694.608975] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9702.547854] Lustre: server umount fs809-OST0000 complete [ 9710.613310] Lustre: server umount fs809-OST0001 complete [ 9720.443519] loop1: detected capacity change from 0 to 204800000 [ 9725.663225] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9725.940763] Lustre: fs809-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9731.790171] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9733.606074] LustreError: 303310:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9733.629323] LustreError: 303310:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 39 previous similar messages [ 9738.031591] loop2: detected capacity change from 0 to 204800000 [ 9742.636309] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9748.729109] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9757.290692] Lustre: server umount fs809-OST0000 complete [ 9766.003659] Lustre: server umount fs809-OST0001 complete [ 9778.657790] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9778.666613] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9778.688714] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9779.691536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9779.693353] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9782.990343] Lustre: server umount lustre-OST0000 complete [ 9788.897486] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9788.904918] LustreError: Skipped 1 previous similar message [ 9788.907823] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9788.933442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9788.943726] Lustre: Skipped 1 previous similar message [ 9791.966375] Lustre: server umount lustre-MDT0000 complete [ 9794.923725] LustreError: 301461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842251 with bad export cookie 11641042480992021086 [ 9794.924805] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9794.931363] LustreError: 301461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9795.289933] Lustre: server umount lustre-MDT0001 complete [ 9804.328753] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 9806.844379] Key type lgssc unregistered [ 9807.105827] LNet: 314725:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9807.111145] LNetError: 314725:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9807.126440] LNet: Removed LNI 192.168.206.108@tcp [ 9807.858465] Key type .llcrypt unregistered [ 9807.863556] Key type ._llcrypt unregistered [ 9836.965564] Key type ._llcrypt registered [ 9836.971210] Key type .llcrypt registered [ 9837.120310] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 9850.024389] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 9851.009344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9851.055444] alg: No test for adler32 (adler32-zlib) [ 9852.153643] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 9852.428901] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 9854.071341] Key type lgssc registered [ 9855.177823] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9888.037341] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9888.048915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9889.243608] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9889.282315] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9889.358817] Lustre: lustre-MDT0000: new disk, initializing [ 9889.428347] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9889.447981] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9893.447845] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9902.263968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9902.362558] Lustre: 318578:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9902.410626] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9902.414350] Lustre: Skipped 1 previous similar message [ 9902.503799] Lustre: lustre-MDT0001: new disk, initializing [ 9902.586993] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9902.632612] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9902.646956] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9906.831917] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9915.295765] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9920.286391] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9926.447939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9926.774406] Lustre: lustre-OST0000: new disk, initializing [ 9926.778921] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9926.785801] Lustre: 320547:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9926.866454] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9932.227385] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9933.913797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9933.932654] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9934.034328] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9940.575445] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9942.531272] Lustre: Mounted lustre-client - version 2.17.57_80_gdb9f866 [ 9943.348482] Lustre: Unmounted lustre-client [ 9952.740668] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9952.755470] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9952.759836] Lustre: Skipped 1 previous similar message [ 9952.784107] Lustre: Skipped 1 previous similar message [ 9957.102115] Lustre: server umount lustre-OST0000 complete [ 9962.981554] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9962.999752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9963.007378] Lustre: Skipped 1 previous similar message [ 9966.602910] Lustre: server umount lustre-MDT0000 complete [ 9970.060674] LustreError: 318572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842426 with bad export cookie 13028707339097028040 [ 9970.078203] LustreError: 318572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9970.082717] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9970.388832] Lustre: server umount lustre-MDT0001 complete [ 9980.775456] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 9984.402278] Key type lgssc unregistered [ 9995.808713] LNet: 1 local NIs in recovery (showing 1): 192.168.206.108@tcp [ 9995.936305] LNet: 322711:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9995.943745] LNetError: 322711:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10073.768853] LNet: Removed LNI 192.168.206.108@tcp [10074.838312] Key type .llcrypt unregistered [10074.841274] Key type ._llcrypt unregistered [10093.066976] Key type ._llcrypt registered [10093.070040] Key type .llcrypt registered [10093.202546] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [10110.751782] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [10112.139835] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10112.222575] alg: No test for adler32 (adler32-zlib) [10114.764709] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [10115.445336] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [10117.239163] Key type lgssc registered [10118.576831] Lustre: Echo OBD driver; http://www.lustre.org/ [10149.555649] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 10:56:46 (1787842606) [10156.070543] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10156.096113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10157.323380] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10157.352981] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10157.477338] Lustre: lustre-MDT0000: new disk, initializing [10157.567892] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10157.616048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10161.994597] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10172.486263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10172.594268] Lustre: 325948:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10172.649797] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10172.656951] Lustre: Skipped 1 previous similar message [10172.717436] Lustre: lustre-MDT0001: new disk, initializing [10172.809469] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10172.842702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10172.858725] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10177.701869] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10186.007833] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10191.145762] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10197.904996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10198.200365] Lustre: lustre-OST0000: new disk, initializing [10198.204425] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10198.213682] Lustre: 327917:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10198.298624] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10204.812124] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10208.366818] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10208.384340] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10208.495766] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10214.666372] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10234.347267] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10234.350948] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10234.361173] Lustre: Skipped 1 previous similar message [10234.368114] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10234.380340] Lustre: Skipped 1 previous similar message [10235.818846] Lustre: server umount lustre-OST0000 complete [10244.581307] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10244.582478] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10244.586712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10244.597795] Lustre: Skipped 1 previous similar message [10246.813249] Lustre: server umount lustre-MDT0000 complete [10249.703708] LustreError: 325958:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10249.728209] LustreError: 325958:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10250.547401] LustreError: 327916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842707 with bad export cookie 18128525171575193656 [10250.557814] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10250.558616] LustreError: 327916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10250.874920] Lustre: server umount lustre-MDT0001 complete [10262.186459] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [10265.236323] Key type lgssc unregistered [10265.528918] LNet: 329826:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10265.542709] LNetError: 329826:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10265.567163] LNet: Removed LNI 192.168.206.108@tcp [10266.381356] Key type .llcrypt unregistered [10266.388930] Key type ._llcrypt unregistered [10287.826566] Key type ._llcrypt registered [10287.828039] Key type .llcrypt registered [10287.943152] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [10289.414784] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10289.451052] alg: No test for adler32 (adler32-zlib) [10290.568805] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [10290.895515] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [10292.655151] Key type lgssc registered [10293.797840] Lustre: Echo OBD driver; http://www.lustre.org/ [10301.831189] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10301.845982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10303.365280] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10308.362634] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10313.698418] LustreError: 331103:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10313.720443] LustreError: 331103:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10317.536364] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10317.798656] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10322.203398] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10329.741545] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10335.552915] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10342.385304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10342.797019] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10349.079988] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10355.701828] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10368.662963] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10370.476479] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10377.703252] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10377.706738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10377.715969] Lustre: Skipped 2 previous similar messages [10377.729376] Lustre: Skipped 2 previous similar messages [10381.842449] Lustre: server umount lustre-MDT0000 complete [10382.817143] LustreError: 331099:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10382.850103] LustreError: 331099:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10385.701140] LustreError: 331083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842842 with bad export cookie 17023194518908679772 [10385.729090] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10387.950414] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10387.954476] LustreError: 332348:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10387.977433] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10387.991434] LustreError: 332348:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10387.995728] Lustre: Skipped 1 previous similar message [10390.948670] LustreError: 332348:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10390.949155] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10390.973608] LustreError: 332348:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10390.996089] Lustre: Skipped 1 previous similar message [10392.140303] Lustre: server umount lustre-MDT0001 complete [10395.832668] Lustre: server umount lustre-OST0000 complete [10404.116495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10408.582514] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10411.514983] Lustre: 335294:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.108@tcp [10413.980760] Lustre: server umount MGS complete [10422.295300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10422.815579] LustreError: 335866:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10422.918360] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10427.548708] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10432.483167] LustreError: 335867:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10432.498207] LustreError: 335867:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10435.211460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10435.542719] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10439.845647] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10448.998739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10449.365061] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10455.421516] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10461.932759] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10463.568591] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [10468.980374] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10470.074896] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10476.000792] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10476.004975] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10476.015827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10478.060908] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10478.061818] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10479.745773] Lustre: server umount lustre-OST0000 complete [10483.170270] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10483.171407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10483.179795] Lustre: Skipped 1 previous similar message [10483.194118] Lustre: Skipped 3 previous similar messages [10489.167979] Lustre: server umount lustre-MDT0000 complete [10492.591056] LustreError: 335846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842949 with bad export cookie 17023194518908680612 [10492.594932] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10492.602217] LustreError: 335846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [10492.873302] Lustre: server umount lustre-MDT0001 complete [10504.020951] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [10507.677895] Key type lgssc unregistered [10508.010164] LNet: 339304:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10508.022196] LNetError: 339304:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10508.058897] LNet: Removed LNI 192.168.206.108@tcp [10509.169930] Key type .llcrypt unregistered [10509.174070] Key type ._llcrypt unregistered [10527.062222] Key type ._llcrypt registered [10527.067691] Key type .llcrypt registered [10527.178994] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [10542.708420] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [10544.149616] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10544.228328] alg: No test for adler32 (adler32-zlib) [10545.428625] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [10545.776872] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [10547.519213] Key type lgssc registered [10548.857181] Lustre: Echo OBD driver; http://www.lustre.org/ [10583.475481] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10583.513296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10584.754335] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10584.780758] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10584.842364] Lustre: lustre-MDT0000: new disk, initializing [10584.913948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10584.926665] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10589.432588] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10599.176200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10599.371404] Lustre: 342467:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10599.473845] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10599.485090] Lustre: Skipped 1 previous similar message [10599.574762] Lustre: lustre-MDT0001: new disk, initializing [10599.658344] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10599.688569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10599.699686] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10604.301848] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10613.046637] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10617.474355] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10623.245575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10623.571936] Lustre: lustre-OST0000: new disk, initializing [10623.574951] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10623.582230] Lustre: 344436:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10623.710806] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10630.205450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10630.214466] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10630.363283] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10630.404731] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10641.179467] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10646.314542] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10646.553547] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10650.969090] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10651.210747] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10656.230257] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10656.231938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10656.238107] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10656.241101] Lustre: Skipped 1 previous similar message [10656.255610] Lustre: Skipped 1 previous similar message [10659.134819] Lustre: server umount lustre-OST0000 complete [10666.469604] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10666.472179] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10666.487898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10666.493411] LustreError: Skipped 1 previous similar message [10666.505885] Lustre: Skipped 1 previous similar message [10668.906746] Lustre: server umount lustre-MDT0000 complete [10671.596205] LustreError: 342479:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10671.629268] LustreError: 342479:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10673.056450] LustreError: 342458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843129 with bad export cookie 16147037643887354494 [10673.059645] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10673.071461] LustreError: 342458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10673.413617] Lustre: server umount lustre-MDT0001 complete [10681.599761] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 11:05:38 (1787843138) [10695.132612] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [10703.775762] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [10734.093879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10734.362806] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10734.381369] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10734.443885] Lustre: lustre-MDT0000: new disk, initializing [10734.547514] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10734.570187] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10738.977928] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10747.990277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10748.089122] Lustre: 349193:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10748.128639] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10748.132225] Lustre: Skipped 1 previous similar message [10748.186608] Lustre: lustre-MDT0001: new disk, initializing [10748.261179] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10748.297035] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10748.310518] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10752.006835] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10760.244123] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10765.826882] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10772.207119] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10772.432504] Lustre: lustre-OST0000: new disk, initializing [10772.436573] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10772.443252] Lustre: 351162:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10772.540712] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10778.354243] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10782.268191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10782.282555] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10782.394087] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10787.070060] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10813.410765] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10813.420318] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10813.427054] Lustre: Skipped 1 previous similar message [10813.435792] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10813.441248] Lustre: Skipped 1 previous similar message [10818.533244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10818.538721] Lustre: Skipped 1 previous similar message [10819.384083] Lustre: server umount lustre-OST0000 complete [10823.648728] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10823.657092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10823.667808] Lustre: Skipped 1 previous similar message [10828.950248] Lustre: server umount lustre-MDT0000 complete [10832.587977] LustreError: 349186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843289 with bad export cookie 16147037643887355376 [10832.603831] LustreError: 349186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10832.608363] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10833.037652] Lustre: server umount lustre-MDT0001 complete [10844.115583] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [10846.808773] Key type lgssc unregistered [10847.133852] LNet: 353318:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10847.150668] LNetError: 353318:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10847.165783] LNet: Removed LNI 192.168.206.108@tcp [10847.879553] Key type .llcrypt unregistered [10847.883753] Key type ._llcrypt unregistered [10930.993437] Key type ._llcrypt registered [10930.996156] Key type .llcrypt registered [10931.074146] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [10931.393287] Key type .llcrypt unregistered [10931.396558] Key type ._llcrypt unregistered [10995.628765] Key type ._llcrypt registered [10995.631183] Key type .llcrypt registered [10995.718992] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [10995.979582] Key type .llcrypt unregistered [10995.983539] Key type ._llcrypt unregistered [11019.348119] Key type ._llcrypt registered [11019.356507] Key type .llcrypt registered [11019.532901] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [11019.947472] Key type .llcrypt unregistered [11019.949271] Key type ._llcrypt unregistered [11047.944907] Key type ._llcrypt registered [11047.947444] Key type .llcrypt registered [11048.053528] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [11048.352536] Key type .llcrypt unregistered [11048.360108] Key type ._llcrypt unregistered [11113.024010] Key type ._llcrypt registered [11113.028086] Key type .llcrypt registered [11113.126265] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [11113.353394] Key type .llcrypt unregistered [11113.359347] Key type ._llcrypt unregistered [11138.242942] Key type ._llcrypt registered [11138.244621] Key type .llcrypt registered [11138.322771] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [11138.617410] Key type .llcrypt unregistered [11138.622176] Key type ._llcrypt unregistered [11159.264609] Key type ._llcrypt registered [11159.268165] Key type .llcrypt registered [11159.339807] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [11159.616371] Key type .llcrypt unregistered [11159.619430] Key type ._llcrypt unregistered [11183.094585] Key type ._llcrypt registered [11183.098184] Key type .llcrypt registered [11183.193280] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [11183.453547] Key type .llcrypt unregistered [11183.455802] Key type ._llcrypt unregistered [11209.112065] Key type ._llcrypt registered [11209.119989] Key type .llcrypt registered [11209.247849] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [11209.615917] Key type .llcrypt unregistered [11209.620289] Key type ._llcrypt unregistered [11235.808363] Key type ._llcrypt registered [11235.812818] Key type .llcrypt registered [11235.936993] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [11236.341693] Key type .llcrypt unregistered [11236.348706] Key type ._llcrypt unregistered [11293.813312] Key type ._llcrypt registered [11293.816082] Key type .llcrypt registered [11293.929816] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [11294.228992] Key type .llcrypt unregistered [11294.231594] Key type ._llcrypt unregistered [11323.795220] Key type ._llcrypt registered [11323.799473] Key type .llcrypt registered [11323.902328] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [11324.208288] Key type .llcrypt unregistered [11324.209898] Key type ._llcrypt unregistered [11349.352123] Key type ._llcrypt registered [11349.353895] Key type .llcrypt registered [11349.418647] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [11349.662157] Key type .llcrypt unregistered [11349.665498] Key type ._llcrypt unregistered [11400.231792] Key type ._llcrypt registered [11400.235845] Key type .llcrypt registered [11400.335227] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [11415.339428] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [11416.512402] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11416.563399] alg: No test for adler32 (adler32-zlib) [11417.637422] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [11417.900080] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [11419.567209] Key type lgssc registered [11420.718600] Lustre: Echo OBD driver; http://www.lustre.org/ [11453.551864] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11453.584154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11454.933779] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [11454.994946] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [11455.085859] Lustre: tst250fs-MDT0000: new disk, initializing [11455.160281] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11455.177667] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11459.302260] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [11469.567852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11469.686271] Lustre: 373481:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11469.763952] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [11469.769413] Lustre: Skipped 1 previous similar message [11469.854375] Lustre: tst250fs-MDT0001: new disk, initializing [11469.998562] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11470.088341] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11470.095594] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11475.234336] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [11480.441731] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11487.479583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11487.741508] Lustre: tst250fs-OST0000: new disk, initializing [11487.747605] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [11487.759209] Lustre: 375118:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11487.852787] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11488.384120] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11488.392890] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11488.483845] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11493.947728] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [11503.940037] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11504.149796] Lustre: tst250fs-OST0001: new disk, initializing [11504.159369] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [11504.168948] Lustre: 375989:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11504.302175] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11510.219276] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [11511.362205] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11511.370890] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11511.437599] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11515.278746] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [11517.718363] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [11517.722885] Lustre: Skipped 2 previous similar messages [11536.866991] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11536.877721] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11536.893258] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [11537.398824] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11537.402060] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [11537.420682] Lustre: Skipped 1 previous similar message [11537.463880] Lustre: Skipped 3 previous similar messages [11541.639181] Lustre: server umount tst250fs-MDT0000 complete [11544.827747] LustreError: 373474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844001 with bad export cookie 16492616767563812755 [11544.836759] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11545.231672] Lustre: server umount tst250fs-MDT0001 complete [11558.650320] Lustre: server umount tst250fs-OST0000 complete [11571.452174] Lustre: server umount tst250fs-OST0001 complete [11587.910530] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [11596.739727] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [11626.480505] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 11:21:23 (1787844083) [11627.790323] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [11644.247912] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [11654.415196] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [11682.789298] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 11416 sec ========= 11:22:19 (1787844139) [11684.499313] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 11:22:21 (1787844141) === [11687.648861] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 11:22:24 (1787844144) === [11724.515104] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [11727.106640] Key type lgssc unregistered [11727.467647] LNet: 385120:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11727.486686] LNetError: 385120:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11727.527572] LNet: Removed LNI 192.168.206.108@tcp [11728.621155] Key type .llcrypt unregistered [11728.622859] Key type ._llcrypt unregistered