[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 513089445 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003318] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.008416] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010015] pid_max: default: 32768 minimum: 301 [ 0.012055] LSM: Security Framework initializing [ 0.013056] Yama: becoming mindful. [ 0.014039] SELinux: Initializing. [ 0.015079] *** VALIDATE selinux *** [ 0.023640] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027828] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028156] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029116] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030122] *** VALIDATE tmpfs *** [ 0.031478] *** VALIDATE proc *** [ 0.032287] *** VALIDATE cgroup *** [ 0.033018] *** VALIDATE cgroup2 *** [ 0.035137] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036183] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038029] Spectre V2 : User space: Vulnerable [ 0.039017] Speculative Store Bypass: Vulnerable [ 0.042245] debug: unmapping init [mem 0xffffffffb0859000-0xffffffffb0860fff] [ 0.044199] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045646] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046037] ... version: 2 [ 0.047018] ... bit width: 48 [ 0.048016] ... generic registers: 4 [ 0.049018] ... value mask: 0000ffffffffffff [ 0.050020] ... max period: 00007fffffffffff [ 0.051018] ... fixed-purpose events: 3 [ 0.052014] ... event mask: 000000070000000f [ 0.053314] rcu: Hierarchical SRCU implementation. [ 0.055539] smp: Bringing up secondary CPUs ... [ 0.056664] x86: Booting SMP configuration: [ 0.057034] .... node #0, CPUs: #1 #2 #3 [ 0.060621] smp: Brought up 1 node, 4 CPUs [ 0.062017] smpboot: Max logical packages: 1 [ 0.063024] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.143872] node 0 deferred pages initialised in 78ms [ 0.147108] devtmpfs: initialized [ 0.148308] x86/mm: Memory block size: 128MB [ 0.151834] gcov: version magic: 0x41383552 [ 0.155283] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.159124] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.161283] pinctrl core: initialized pinctrl subsystem [ 0.164235] [ 0.164706] ************************************************************* [ 0.167017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.169015] ** ** [ 0.172013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.174014] ** ** [ 0.177017] ** This means that this kernel is built to expose internal ** [ 0.179012] ** IOMMU data structures, which may compromise security on ** [ 0.182016] ** your system. ** [ 0.184012] ** ** [ 0.187019] ** If you see this message and you are not debugging the ** [ 0.189011] ** kernel, report this immediately to your vendor! ** [ 0.190010] ** ** [ 0.192013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.194056] ************************************************************* [ 0.197676] NET: Registered protocol family 16 [ 0.199492] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.202073] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.204045] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.208105] cpuidle: using governor menu [ 0.209967] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.213786] PCI: Using configuration type 1 for base access [ 0.216155] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.227140] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.230051] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.234092] cryptd: max_cpu_qlen set to 1000 [ 0.237270] ACPI: Added _OSI(Module Device) [ 0.238018] ACPI: Added _OSI(Processor Device) [ 0.240018] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.241015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.247942] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.255197] ACPI: Interpreter enabled [ 0.256061] ACPI: PM: (supports S0 S3 S4 S5) [ 0.258016] ACPI: Using IOAPIC for interrupt routing [ 0.260117] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.263423] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.272746] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.275048] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.277021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.280163] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.286000] acpiphp: Slot [2] registered [ 0.287096] acpiphp: Slot [5] registered [ 0.289148] acpiphp: Slot [6] registered [ 0.290107] acpiphp: Slot [7] registered [ 0.291110] acpiphp: Slot [8] registered [ 0.292188] acpiphp: Slot [9] registered [ 0.293141] acpiphp: Slot [10] registered [ 0.294042] acpiphp: Slot [3] registered [ 0.295114] acpiphp: Slot [4] registered [ 0.297103] acpiphp: Slot [11] registered [ 0.299114] acpiphp: Slot [12] registered [ 0.300131] acpiphp: Slot [13] registered [ 0.302123] acpiphp: Slot [14] registered [ 0.303100] acpiphp: Slot [15] registered [ 0.305116] acpiphp: Slot [16] registered [ 0.306114] acpiphp: Slot [17] registered [ 0.308122] acpiphp: Slot [18] registered [ 0.309106] acpiphp: Slot [19] registered [ 0.311130] acpiphp: Slot [20] registered [ 0.312121] acpiphp: Slot [21] registered [ 0.314168] acpiphp: Slot [22] registered [ 0.316128] acpiphp: Slot [23] registered [ 0.317115] acpiphp: Slot [24] registered [ 0.320119] acpiphp: Slot [25] registered [ 0.321124] acpiphp: Slot [26] registered [ 0.322141] acpiphp: Slot [27] registered [ 0.323000] acpiphp: Slot [28] registered [ 0.323000] acpiphp: Slot [29] registered [ 0.325109] acpiphp: Slot [30] registered [ 0.327144] acpiphp: Slot [31] registered [ 0.329080] PCI host bridge to bus 0000:00 [ 0.330020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.332023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.335021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.337023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.340027] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.343028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.345190] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.349197] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.352583] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.362880] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.366000] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.368019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.371020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.374023] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.377647] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.379839] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.383061] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.387572] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.393019] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.408018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.413000] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.418991] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.430015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.437018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.465015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.478187] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.485015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.492015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.509020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.519587] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.528017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.535017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.556021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.565000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.574016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.582019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.600015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.611724] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.620020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.628024] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.648019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.658000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.669018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.677018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.693014] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.706693] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.708425] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.711387] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.713270] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.715359] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.721127] iommu: Default domain type: Passthrough [ 0.722476] SCSI subsystem initialized [ 0.724126] ACPI: bus type USB registered [ 0.726105] usbcore: registered new interface driver usbfs [ 0.727079] usbcore: registered new interface driver hub [ 0.730105] usbcore: registered new device driver usb [ 0.731140] pps_core: LinuxPPS API ver. 1 registered [ 0.733010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.735062] PTP clock support registered [ 0.737073] EDAC MC: Ver: 3.0.0 [ 0.738469] PCI: Using ACPI for IRQ routing [ 0.740553] NetLabel: Initializing [ 0.742013] NetLabel: domain hash size = 128 [ 0.744013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.745081] NetLabel: unlabeled traffic allowed by default [ 0.748190] vgaarb: loaded [ 0.750205] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.752015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.758006] clocksource: Switched to clocksource kvm-clock [ 0.866742] VFS: Disk quotas dquot_6.6.0 [ 0.869930] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.873105] *** VALIDATE ramfs *** [ 0.874552] *** VALIDATE hugetlbfs *** [ 0.876273] pnp: PnP ACPI init [ 0.878668] pnp: PnP ACPI: found 6 devices [ 0.898883] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.902853] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.905353] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.908083] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.911039] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.913449] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.917155] NET: Registered protocol family 2 [ 0.919661] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.925126] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.929380] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.934838] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.938471] TCP: Hash tables configured (established 65536 bind 65536) [ 0.941329] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.944646] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.947690] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.950475] NET: Registered protocol family 1 [ 0.954301] RPC: Registered named UNIX socket transport module. [ 0.956963] RPC: Registered udp transport module. [ 0.959090] RPC: Registered tcp transport module. [ 0.960942] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.963728] NET: Registered protocol family 44 [ 0.965505] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.967900] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.970221] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.972718] PCI: CLS 0 bytes, default 64 [ 0.974729] Unpacking initramfs... [ 2.402353] debug: unmapping init [mem 0xffffa0cd3cc54000-0xffffa0cd3ffbffff] [ 2.409504] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.411987] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.415343] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.971602] Initialise system trusted keyrings [ 2.974229] Key type blacklist registered [ 2.979196] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.990596] zbud: loaded [ 2.994267] *** VALIDATE nfs *** [ 2.995607] *** VALIDATE nfs4 *** [ 2.997861] pstore: using deflate compression [ 3.002234] Platform Keyring initialized [ 3.107096] NET: Registered protocol family 38 [ 3.108818] Key type asymmetric registered [ 3.110375] Asymmetric key parser 'x509' registered [ 3.112240] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.116214] io scheduler mq-deadline registered [ 3.117520] io scheduler kyber registered [ 3.118888] io scheduler bfq registered [ 3.120709] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.123573] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.126956] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.130331] ACPI: Power Button [PWRF] [ 3.137604] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.145450] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.161287] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.173744] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.192873] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.219770] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.248723] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.255250] Non-volatile memory driver v1.3 [ 3.256872] Linux agpgart interface v0.103 [ 3.294736] virtio_blk virtio1: [vda] 149760 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.297899] vda: detected capacity change from 0 to 76677120 [ 3.312525] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.315922] vdb: detected capacity change from 0 to 1073741824 [ 3.334404] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.339767] vdc: detected capacity change from 0 to 2621440000 [ 3.354393] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.357566] vdd: detected capacity change from 0 to 2621440000 [ 3.382437] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.385656] vde: detected capacity change from 0 to 4294967296 [ 3.403216] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.406614] vdf: detected capacity change from 0 to 4294967296 [ 3.416124] libphy: Fixed MDIO Bus: probed [ 3.421954] usbcore: registered new interface driver usbserial_generic [ 3.424434] usbserial: USB Serial support registered for generic [ 3.426641] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.431157] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.433528] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.436347] mousedev: PS/2 mouse device common for all mice [ 3.439359] rtc_cmos 00:05: RTC can wake from S4 [ 3.442443] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.446488] rtc_cmos 00:05: registered as rtc0 [ 3.448583] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.449226] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.454741] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.455074] intel_pstate: CPU model not supported [ 3.463690] hid: raw HID events driver (C) Jiri Kosina [ 3.466256] usbcore: registered new interface driver usbhid [ 3.468884] usbhid: USB HID core driver [ 3.470832] drop_monitor: Initializing network drop monitor service [ 3.473673] Initializing XFRM netlink socket [ 3.475560] NET: Registered protocol family 10 [ 3.479366] Segment Routing with IPv6 [ 3.480664] NET: Registered protocol family 17 [ 3.482871] mpls_gso: MPLS GSO support [ 3.491948] RAS: Correctable Errors collector initialized. [ 3.494372] AVX version of gcm_enc/dec engaged. [ 3.496174] AES CTR mode by8 optimization enabled [ 3.582598] sched_clock: Marking stable (3582564922, 0)->(4540439061, -957874139) [ 3.587224] registered taskstats version 1 [ 3.589697] Loading compiled-in X.509 certificates [ 3.592185] zswap: loaded using pool lzo/zbud [ 3.621392] Key type big_key registered [ 3.634221] Key type encrypted registered [ 3.635879] ima: No TPM chip found, activating TPM-bypass! [ 3.638244] ima: Allocated hash algorithm: sha1 [ 3.640047] ima: No architecture policies found [ 3.641654] evm: Initialising EVM extended attributes: [ 3.643746] evm: security.selinux [ 3.645139] evm: security.ima [ 3.646328] evm: security.capability [ 3.647736] evm: HMAC attrs: 0x1 [ 3.650489] rtc_cmos 00:05: setting system clock to 2026-09-03 12:12:07 UTC (1788437527) [ 3.657369] debug: unmapping init [mem 0xffffffffb1803000-0xffffffffb19fffff] [ 3.660710] debug: unmapping init [mem 0xffffffffb0582000-0xffffffffb0858fff] [ 3.668094] Write protecting the kernel read-only data: 28672k [ 3.671171] debug: unmapping init [mem 0xffffffffaec03000-0xffffffffaedfffff] [ 3.674355] debug: unmapping init [mem 0xffffffffaf514000-0xffffffffaf5fffff] [ 3.712940] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.721686] systemd[1]: Detected virtualization kvm. [ 3.724246] systemd[1]: Detected architecture x86-64. [ 3.725979] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.754520] systemd[1]: No hostname configured. [ 3.756289] systemd[1]: Set hostname to . [ 3.758390] random: systemd: uninitialized urandom read (16 bytes read) [ 3.760817] systemd[1]: Initializing machine ID from random generator. [ 3.916844] random: systemd: uninitialized urandom read (16 bytes read) [ 3.919417] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.924275] random: systemd: uninitialized urandom read (16 bytes read) [ 3.927388] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.931239] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.586477] device-mapper: uevent: version 1.0.3 [ 4.588601] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.409772] virtio_net virtio0 ens2: renamed from eth0 [ 5.415109] random: fast init done [ 5.428752] scsi host0: ata_piix [ 5.499259] scsi host1: ata_piix [ 5.502459] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.505174] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.913632] dracut-initqueue[598]: RTNETLINK answers: File exists [ 10.153230] random: crng init done [ 10.154876] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.694988] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.851989] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.131872] SELinux: Disabled at runtime. [ 12.194071] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.205483] systemd[1]: Detected virtualization kvm. [ 12.207707] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.762060] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.765788] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.772110] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.776331] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.780080] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.787223] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.791310] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Stopped target Switch Root. Mounting Kernel Debug File System... [ OK ] Listening on initctl Compatibility Named Pipe. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Huge Pages File System... [ 12.844590] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target rpc_pipefs.target. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Initrd Root File System. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.237099] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.492489] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.547909] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.679506] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.699600] EDAC sbridge: Ver: 1.1.2 [ 15.400506] Key type dns_resolver registered [ 15.716679] NFS: Registering the id_resolver key type [ 15.718771] Key type id_resolver registered [ 15.720475] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg456-server login: [ 37.827032] hrtimer: interrupt took 2089302 ns [ 49.699360] libcfs: loading out-of-tree module taints kernel. [ 49.759931] Key type ._llcrypt registered [ 49.763996] Key type .llcrypt registered [ 49.858925] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [ 72.885482] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 74.703766] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 74.738280] alg: No test for adler32 (adler32-zlib) [ 76.213485] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 76.887572] LNet: Added LNI 192.168.204.156@tcp [8/256/0/180] [ 78.719187] Key type lgssc registered [ 80.836373] Lustre: Echo OBD driver; http://www.lustre.org/ [ 97.881597] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 138.244505] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 151.170957] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 151.217464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 152.648835] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 152.718308] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 152.906629] Lustre: lustre-MDT0000: new disk, initializing [ 153.034667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 153.071612] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 157.820640] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 172.516525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 172.638499] Lustre: 6524:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 172.680203] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 172.683645] Lustre: Skipped 1 previous similar message [ 172.742378] Lustre: lustre-MDT0001: new disk, initializing [ 172.817738] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 172.853883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 172.860164] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 177.572819] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 182.526906] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 192.060141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 192.338584] Lustre: lustre-OST0000: new disk, initializing [ 192.343048] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 192.355753] Lustre: 8462:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 192.428692] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 194.401238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 194.413392] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 194.563123] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 198.506824] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 213.530234] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 213.688382] Lustre: lustre-OST0001: new disk, initializing [ 213.690823] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 213.699929] Lustre: 9535:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 213.764591] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 220.245432] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 220.260489] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 220.393903] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 220.714138] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 233.517919] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 238.393935] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 244.791982] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing check_logdir /tmp/testlogs/ [ 249.720318] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing yml_node [ 253.751782] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 256.652971] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 259.292263] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 261.124937] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 3 08:16:22 EDT 2026 [ 279.162997] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 280.689879] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 297.443606] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 297.446707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 297.455555] Lustre: Skipped 3 previous similar messages [ 297.461594] Lustre: Skipped 2 previous similar messages [ 302.565084] Lustre: server umount lustre-MDT0000 complete [ 306.374332] LustreError: 6517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788437830 with bad export cookie 2500927334833144655 [ 306.375801] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 306.385946] LustreError: 6517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 307.680943] LustreError: 11392:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 307.681776] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 307.683026] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 307.683032] Lustre: Skipped 1 previous similar message [ 307.703780] LustreError: 11392:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 307.754062] Lustre: Skipped 1 previous similar message [ 311.712238] LustreError: 11392:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 311.712835] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 311.728613] LustreError: 11392:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 311.750990] Lustre: Skipped 2 previous similar messages [ 312.832444] Lustre: server umount lustre-MDT0001 complete [ 316.714933] Lustre: server umount lustre-OST0000 complete [ 320.647418] Lustre: server umount lustre-OST0001 complete [ 326.815869] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [ 335.634555] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 367.793460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 368.036940] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 368.060108] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 368.150161] Lustre: lustre-MDT0000: new disk, initializing [ 368.264532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 368.280148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 373.339899] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 387.544150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 387.686334] Lustre: 15090:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 387.805720] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 387.815161] Lustre: Skipped 1 previous similar message [ 387.963931] Lustre: lustre-MDT0001: new disk, initializing [ 388.151359] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 388.300834] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 388.331879] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 395.660920] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 407.671384] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 416.003377] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 424.820974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 425.143382] Lustre: lustre-OST0000: new disk, initializing [ 425.148755] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 425.158322] Lustre: 17063:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 425.306131] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 426.303806] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 426.321520] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 426.509864] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 431.689490] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 440.320354] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 444.694601] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 444.929790] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 449.928632] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 450.166406] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 457.190563] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 457.197600] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 457.199916] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 457.202073] LustreError: Skipped 1 previous similar message [ 457.227641] Lustre: Skipped 1 previous similar message [ 458.535665] Lustre: server umount lustre-OST0000 complete [ 462.306292] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 462.309430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 462.320851] Lustre: Skipped 1 previous similar message [ 462.335363] Lustre: Skipped 2 previous similar messages [ 468.574711] Lustre: server umount lustre-MDT0000 complete [ 472.181930] LustreError: 15081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788437996 with bad export cookie 2500927334833146188 [ 472.183563] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 472.190791] LustreError: 15081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 472.496332] Lustre: server umount lustre-MDT0001 complete [ 477.840752] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 08:19:59 (1788437999) [ 496.111043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 496.662257] LustreError: 19936:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 496.731392] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 501.043118] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 501.732635] LustreError: 19937:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 506.855338] LustreError: 19936:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 511.518087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 518.066409] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 527.446632] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 534.368589] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 536.833759] Lustre: Failing over lustre-MDT0000 [ 537.206293] Lustre: server umount lustre-MDT0000 complete [ 537.575756] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 537.587785] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 537.609119] LustreError: 19936:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 557.039492] LustreError: 19932:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 557.052632] LustreError: 19932:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 557.098153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 557.259916] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 562.579762] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 562.668560] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 562.674757] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 562.706613] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 572.226905] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 576.015420] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 576.022279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 576.025065] Lustre: Skipped 2 previous similar messages [ 576.034447] Lustre: Skipped 3 previous similar messages [ 590.303148] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 590.471186] Lustre: server umount lustre-MDT0000 complete [ 592.354712] LustreError: 19937:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 592.382746] LustreError: 19937:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 594.118867] LustreError: 19917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438117 with bad export cookie 2500927334833147602 [ 594.138169] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 594.142850] LustreError: 19917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 594.596248] Lustre: server umount lustre-MDT0001 complete [ 602.723324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 603.234300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 603.237928] Lustre: Skipped 2 previous similar messages [ 607.243480] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 615.421863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 619.964197] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 627.258430] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 633.444568] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 635.597802] Lustre: Failing over lustre-MDT0000 [ 635.785863] Lustre: server umount lustre-MDT0000 complete [ 636.385913] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 636.395278] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 654.818440] Lustre: 3660:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788438162/real 1788438162] req@ffffa0cdb79fa680 x1875312745044224/t0(0) o400->MGC192.168.204.156@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788438178 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 654.859630] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 655.086514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 659.938450] LustreError: 23276:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 659.960847] LustreError: 23276:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 665.066388] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x22b5142919cb527c [ 665.090153] Lustre: MGC192.168.204.156@tcp: Connection restored to 0@lo (at 0@lo) [ 665.104385] Lustre: Skipped 1 previous similar message [ 665.348684] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 670.645503] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 670.696497] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 670.710361] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 670.753215] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 679.813851] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 685.157145] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 685.182385] Lustre: Skipped 1 previous similar message [ 685.198554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 685.204248] Lustre: Skipped 7 previous similar messages [ 690.352244] Lustre: server umount lustre-MDT0000 complete [ 694.000203] LustreError: 23257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438217 with bad export cookie 2500927334833148540 [ 694.010596] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 694.017084] LustreError: 23257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 694.379313] Lustre: server umount lustre-MDT0001 complete [ 703.165502] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 08:23:44 (1788438224) [ 720.556348] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [ 729.818186] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 766.620864] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 777.844312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 778.240773] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 778.276677] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 778.371326] Lustre: lustre-MDT0000: new disk, initializing [ 778.493219] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 778.503123] Lustre: Skipped 2 previous similar messages [ 778.537733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 783.496836] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 794.398254] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 794.497727] Lustre: 29650:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 794.789733] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 799.482977] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 809.760679] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 815.971386] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 823.815099] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 824.095572] Lustre: 31619:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 825.440993] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 825.486965] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 825.572202] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 825.605804] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 830.715347] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 840.064035] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 846.009194] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 871.906895] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 871.908990] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 871.926295] Lustre: Skipped 1 previous similar message [ 871.944781] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 871.948342] Lustre: Skipped 1 previous similar message [ 882.147369] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 882.465892] Lustre: server umount lustre-OST0000 complete [ 886.762832] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 886.769351] LustreError: Skipped 1 previous similar message [ 901.087147] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 901.298827] Lustre: server umount lustre-MDT0000 complete [ 902.624204] LustreError: 29663:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 902.640262] LustreError: 29663:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 905.275990] LustreError: 29643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438429 with bad export cookie 2500927334833148946 [ 905.284289] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 905.294669] LustreError: 29643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 905.706167] Lustre: server umount lustre-MDT0001 complete [ 917.111230] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [ 920.235581] Key type lgssc unregistered [ 920.582937] LNet: 33601:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 920.595187] LNetError: 33601:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 920.612289] LNet: Removed LNI 192.168.204.156@tcp [ 921.663163] Key type .llcrypt unregistered [ 921.664981] Key type ._llcrypt unregistered [ 948.534546] Key type ._llcrypt registered [ 948.536711] Key type .llcrypt registered [ 948.669355] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 949.615685] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 949.635861] alg: No test for adler32 (adler32-zlib) [ 950.970654] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 951.347960] LNet: Added LNI 192.168.204.156@tcp [8/256/0/180] [ 953.063223] Key type lgssc registered [ 954.272611] Lustre: Echo OBD driver; http://www.lustre.org/ [ 962.497948] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 962.515576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 964.089547] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 968.913545] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 973.301481] LustreError: 35013:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 973.316274] LustreError: 35013:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 978.290945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 978.404204] LustreError: 35012:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 978.667422] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 983.518912] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 993.048318] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 999.540564] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1008.048875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1008.506258] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1017.144874] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1024.890576] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1027.225583] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1042.289828] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 08:29:23 (1788438563) [ 1044.560168] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1046.545298] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1053.065356] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 08:29:34 (1788438574) [ 1061.719672] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 08:29:43 (1788438583) [ 1062.720048] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1070.050744] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1070.057519] Lustre: Skipped 2 previous similar messages [ 1071.584319] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 08:29:53 (1788438593) [ 1074.811733] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1074.819754] Lustre: Skipped 3 previous similar messages [ 1090.485692] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1090.491227] Lustre: Skipped 6 previous similar messages [ 1092.785627] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 08:30:14 (1788438614) [ 1095.246411] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1098.446475] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1105.755784] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 08:30:27 (1788438627) [ 1107.792681] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1107.796161] Lustre: Skipped 3 previous similar messages [ 1120.401342] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 08:30:41 (1788438641) [ 1137.644515] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1137.647793] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1137.656264] LustreError: Skipped 1 previous similar message [ 1137.656604] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1137.693437] Lustre: Skipped 1 previous similar message [ 1139.719682] Lustre: server umount lustre-OST0000 complete [ 1147.872861] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1147.874912] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1147.880892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1147.880900] Lustre: Skipped 1 previous similar message [ 1147.883101] Lustre: Skipped 1 previous similar message [ 1149.993278] Lustre: server umount lustre-MDT0000 complete [ 1152.993273] LustreError: 36702:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1153.021602] LustreError: 36702:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1154.221227] LustreError: 34994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438678 with bad export cookie 3882570596618226883 [ 1154.231965] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1154.232791] LustreError: 34994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1154.786655] Lustre: server umount lustre-MDT0001 complete [ 1165.669672] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [ 1168.251461] Key type lgssc unregistered [ 1168.542711] LNet: 42860:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1168.549318] LNetError: 42860:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1168.562690] LNet: Removed LNI 192.168.204.156@tcp [ 1169.485488] Key type .llcrypt unregistered [ 1169.487317] Key type ._llcrypt unregistered [ 1186.252726] Key type ._llcrypt registered [ 1186.254976] Key type .llcrypt registered [ 1186.352498] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [ 1200.217932] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 1201.377971] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1201.452347] alg: No test for adler32 (adler32-zlib) [ 1202.528281] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1202.793241] LNet: Added LNI 192.168.204.156@tcp [8/256/0/180] [ 1204.487162] Key type lgssc registered [ 1205.762972] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1238.351520] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1238.389302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1239.591680] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1239.639336] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1239.722369] Lustre: lustre-MDT0000: new disk, initializing [ 1239.860241] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1239.895499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1243.872529] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1254.416038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1254.493366] Lustre: 46009:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1254.547501] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1254.551714] Lustre: Skipped 1 previous similar message [ 1254.623115] Lustre: lustre-MDT0001: new disk, initializing [ 1254.719304] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1254.749436] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1254.769804] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1259.168805] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1268.138869] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1273.688450] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1280.350775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1280.597791] Lustre: lustre-OST0000: new disk, initializing [ 1280.602378] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1280.609783] Lustre: 47979:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1280.687059] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1286.203823] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1288.771908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1288.778468] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1288.913332] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1295.333169] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1300.320742] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1300.539447] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1304.934305] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1305.140717] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1309.664436] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1309.667078] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1309.676711] LustreError: Skipped 1 previous similar message [ 1309.681727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1309.687293] Lustre: Skipped 1 previous similar message [ 1309.714410] Lustre: Skipped 1 previous similar message [ 1312.965792] Lustre: server umount lustre-OST0000 complete [ 1319.906455] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1319.913416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1319.932365] Lustre: Skipped 1 previous similar message [ 1319.952179] Lustre: Skipped 1 previous similar message [ 1322.991637] Lustre: server umount lustre-MDT0000 complete [ 1327.542728] LustreError: 46000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438851 with bad export cookie 13654633157988656621 [ 1327.554205] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1327.561785] LustreError: 46000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1328.108149] Lustre: server umount lustre-MDT0001 complete [ 1339.272595] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 08:34:20 (1788438860) [ 1346.156341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1346.567908] LustreError: 50441:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1346.590575] LustreError: 50441:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1346.665167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1351.179461] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1351.651384] LustreError: 50442:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1356.770729] LustreError: 50441:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1360.892625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1361.407570] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1365.839412] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1373.669738] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1380.129390] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1387.842824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1388.316403] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1394.702388] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1403.110333] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1410.795767] LustreError: 52970:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1411.789923] Lustre: Modifying parameter general.timeout=1 in log params [ 1412.890379] Lustre: Modifying parameter general.timeout=2 in log params [ 1415.318316] Lustre: Modifying parameter general.timeout=5 in log params [ 1415.324399] Lustre: Skipped 2 previous similar messages [ 1419.466274] Lustre: Modifying parameter general.timeout=10 in log params [ 1419.471335] Lustre: Skipped 4 previous similar messages [ 1428.410245] Lustre: Modifying parameter general.timeout=20 in log params [ 1428.418990] Lustre: Skipped 9 previous similar messages [ 1444.566180] Lustre: Modifying parameter general.timeout=36 in log params [ 1444.572685] Lustre: Skipped 15 previous similar messages [ 1476.833078] Lustre: Modifying parameter general.timeout=69 in log params [ 1476.835071] Lustre: Skipped 32 previous similar messages [ 1540.773337] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 08:37:42 (1788439062) [ 1541.592222] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_policies=tbf in log params [ 1541.597465] Lustre: Skipped 61 previous similar messages [ 1543.469687] Lustre: 59913:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1545.132933] Lustre: 60022:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1548.896795] Lustre: 60219:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1548.907576] Lustre: 60219:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1581.723245] Lustre: 61789:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1581.735770] Lustre: 61789:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1581.763196] LustreError: 61789:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1586.176578] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 08:38:27 (1788439107) [ 1600.995317] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1600.996684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1601.005487] Lustre: Skipped 1 previous similar message [ 1605.649863] Lustre: server umount lustre-MDT0000 complete [ 1606.114129] LustreError: 51169:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1606.132385] LustreError: 51169:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1609.040995] LustreError: 50423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439132 with bad export cookie 13654633157988657503 [ 1609.050257] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1609.053260] LustreError: 50423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 1609.434065] Lustre: server umount lustre-MDT0001 complete [ 1623.807186] Lustre: server umount lustre-OST0000 complete [ 1651.795412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1652.068410] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1652.453359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1657.566384] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1667.702956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1667.796106] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1667.830788] Lustre: 64320:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1673.887410] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1683.232123] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1689.881108] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1697.168966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1697.425589] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1697.665211] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1697.673777] Lustre: Skipped 1 previous similar message [ 1704.878484] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1712.907789] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1714.712673] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1714.728532] Lustre: Skipped 22 previous similar messages [ 1714.782868] Lustre: 66888:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1726.946290] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1726.963819] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1726.977338] Lustre: Skipped 1 previous similar message [ 1726.989824] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1726.998636] Lustre: Skipped 2 previous similar messages [ 1729.775172] Lustre: server umount lustre-OST0000 complete [ 1734.625041] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1734.641976] LustreError: Skipped 1 previous similar message [ 1734.651567] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1734.679976] Lustre: Skipped 1 previous similar message [ 1734.692369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1734.694978] Lustre: Skipped 2 previous similar messages [ 1740.434424] Lustre: server umount lustre-MDT0000 complete [ 1742.837117] LustreError: 64332:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1742.863995] LustreError: 64332:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1745.123668] LustreError: 65064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439268 with bad export cookie 13654633157988660520 [ 1745.137896] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1745.139647] LustreError: 65064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1745.653863] Lustre: server umount lustre-MDT0001 complete [ 1756.917679] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [ 1759.466888] Key type lgssc unregistered [ 1759.738620] LNet: 68093:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1759.753755] LNetError: 68093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1759.776659] LNet: Removed LNI 192.168.204.156@tcp [ 1760.617864] Key type .llcrypt unregistered [ 1760.623625] Key type ._llcrypt unregistered [ 1785.813928] Key type ._llcrypt registered [ 1785.819489] Key type .llcrypt registered [ 1785.979735] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 1787.285825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1787.313664] alg: No test for adler32 (adler32-zlib) [ 1788.413228] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1788.680724] LNet: Added LNI 192.168.204.156@tcp [8/256/0/180] [ 1790.367396] Key type lgssc registered [ 1791.574043] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1798.416764] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1798.438431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1799.904175] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1805.521568] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1809.384183] LustreError: 69503:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1809.418872] LustreError: 69503:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1814.507792] LustreError: 69504:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1818.281290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1819.008408] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1825.057854] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1833.103426] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1839.349958] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1846.598940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1847.001863] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1852.759379] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1861.633349] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1880.504294] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1890.275925] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1890.282680] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1890.304378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1891.297423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1891.301403] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1891.315875] Lustre: Skipped 1 previous similar message [ 1891.345226] Lustre: Skipped 1 previous similar message [ 1896.417191] LustreError: 69504:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1896.433549] LustreError: 69504:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1896.482480] Lustre: server umount lustre-MDT0000 complete [ 1901.327466] LustreError: 69484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439425 with bad export cookie 14297890062575322561 [ 1901.338673] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1901.540966] LustreError: 69498:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1901.554708] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1901.973330] Lustre: server umount lustre-MDT0001 complete [ 1908.999350] Lustre: server umount lustre-OST0000 complete [ 1942.473358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1942.700699] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1942.752405] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1943.232750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1948.580724] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1959.349520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1959.534625] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1959.583506] Lustre: 74399:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1960.407642] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1967.065685] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1977.452571] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1985.701777] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1996.080072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1996.367875] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1996.881101] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2004.916523] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 2014.562310] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2016.888663] Lustre: 76975:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2019.278084] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2029.028798] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2029.042975] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2029.065235] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2029.074476] Lustre: Skipped 1 previous similar message [ 2031.936800] Lustre: server umount lustre-OST0000 complete [ 2036.704199] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2036.715969] LustreError: Skipped 1 previous similar message [ 2036.719194] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2036.736977] Lustre: Skipped 1 previous similar message [ 2036.742932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2036.750137] Lustre: Skipped 1 previous similar message [ 2042.468774] Lustre: server umount lustre-MDT0000 complete [ 2044.904925] LustreError: 74410:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2044.923555] LustreError: 74410:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2046.801994] LustreError: 74391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439570 with bad export cookie 14297890062575323415 [ 2046.805097] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2046.807636] LustreError: 74391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2047.121409] Lustre: server umount lustre-MDT0001 complete [ 2063.259256] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [ 2068.433904] Key type lgssc unregistered [ 2068.957237] LNet: 78190:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2068.970185] LNetError: 78190:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2068.989940] LNet: Removed LNI 192.168.204.156@tcp [ 2070.433593] Key type .llcrypt unregistered [ 2070.438302] Key type ._llcrypt unregistered [ 2098.478513] Key type ._llcrypt registered [ 2098.481379] Key type .llcrypt registered [ 2098.581531] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 2099.972386] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2100.160776] alg: No test for adler32 (adler32-zlib) [ 2101.479573] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2101.727400] LNet: Added LNI 192.168.204.156@tcp [8/256/0/180] [ 2103.447221] Key type lgssc registered [ 2104.675792] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2114.197603] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2114.224336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2115.773194] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2121.393780] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 2125.283349] LustreError: 79599:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2125.304409] LustreError: 79599:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2130.407675] LustreError: 79598:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2132.072383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2132.917926] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2139.260704] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 2148.399962] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2154.704347] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2162.460446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2163.098132] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2171.031851] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 2180.599570] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2194.055620] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2195.144621] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2196.262894] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2198.341882] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2198.348934] Lustre: Skipped 1 previous similar message [ 2202.362920] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2202.368597] Lustre: Skipped 3 previous similar messages [ 2210.698344] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2210.708659] Lustre: Skipped 7 previous similar messages [ 2226.760632] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2226.766668] Lustre: Skipped 14 previous similar messages [ 2259.067721] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2259.075966] Lustre: Skipped 28 previous similar messages [ 2312.984177] Lustre: 87739:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2315.962329] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 08:50:37 (1788439837) [ 2317.996104] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2320.477780] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 08:50:41 (1788439841) [ 2324.965767] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2324.969394] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2324.975633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2330.084178] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2330.091389] Lustre: Skipped 2 previous similar messages [ 2335.205706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2335.215980] Lustre: Skipped 1 previous similar message [ 2339.298301] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2339.644651] Lustre: server umount lustre-OST0000 complete [ 2345.442905] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2345.461186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2345.468720] Lustre: Skipped 2 previous similar messages [ 2345.484981] Lustre: Skipped 1 previous similar message [ 2350.593120] LustreError: 80332:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2350.603718] LustreError: 80332:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2350.647427] Lustre: server umount lustre-MDT0000 complete [ 2355.534585] LustreError: 79580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439879 with bad export cookie 11339440804649501048 [ 2355.543236] LustreError: 79580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2355.545131] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2356.252990] Lustre: server umount lustre-MDT0001 complete [ 2369.712808] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [ 2373.512629] Key type lgssc unregistered [ 2373.861731] LNet: 88988:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2373.872079] LNetError: 88988:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2373.896569] LNet: Removed LNI 192.168.204.156@tcp [ 2374.760709] Key type .llcrypt unregistered [ 2374.766395] Key type ._llcrypt unregistered [ 2380.394611] Key type ._llcrypt registered [ 2380.398360] Key type .llcrypt registered [ 2380.539746] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2387.021098] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules [ 2388.098684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2388.267460] alg: No test for adler32 (adler32-zlib) [ 2389.524362] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2389.889526] LNet: Added LNI 192.168.204.156@tcp [8/256/0/180] [ 2391.609642] Key type lgssc registered [ 2393.535891] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2395.277427] LustreError: 90234:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2416.831701] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 2417.601086] LustreError: 90234:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2426.228329] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2426.260687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2427.901909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2434.879741] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 2438.116018] LustreError: 91259:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2438.141646] LustreError: 91259:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2443.240131] LustreError: 91260:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2447.577459] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 08:52:48 (1788439968) [ 2448.354400] LustreError: 91259:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2452.490991] LustreError: 91260:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2453.364951] LustreError: 91291:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2453.385695] LustreError: 91291:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2453.395871] LustreError: 91291:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 26, retries 0, failed: rc = -5 [ 2453.930507] Lustre: server umount lustre-MDT0000 complete [ 2469.390055] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [ 2471.891525] Key type lgssc unregistered [ 2472.256371] LNet: 92784:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2472.268609] LNetError: 92784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2472.291333] LNet: Removed LNI 192.168.204.156@tcp [ 2473.330581] Key type .llcrypt unregistered [ 2473.334027] Key type ._llcrypt unregistered [ 2493.032242] Key type ._llcrypt registered [ 2493.034507] Key type .llcrypt registered [ 2493.132726] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 2494.491421] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2494.825460] alg: No test for adler32 (adler32-zlib) [ 2495.952839] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2496.263142] LNet: Added LNI 192.168.204.156@tcp [8/256/0/180] [ 2498.031378] Key type lgssc registered [ 2499.367699] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2508.437207] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2508.452741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2510.212665] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2516.048402] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 2519.524557] LustreError: 93998:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2519.538730] LustreError: 93998:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2524.641360] LustreError: 93999:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2525.379579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2526.036037] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2530.942310] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 2539.942812] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2545.958580] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2554.510169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2554.803040] LustreError: 95827:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2555.336787] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2562.670464] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 2572.549818] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2633.332771] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 08:55:55 (1788440155) [ 2635.593389] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2638.517567] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 08:55:59 (1788440159) [ 2647.525181] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2647.542945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2647.545239] Lustre: Skipped 2 previous similar messages [ 2647.563727] Lustre: Skipped 2 previous similar messages [ 2650.109912] Lustre: server umount lustre-MDT0000 complete [ 2652.642927] LustreError: 93999:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2652.667638] LustreError: 93999:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2654.606353] LustreError: 93979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440178 with bad export cookie 14103138549404227235 [ 2654.615533] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2654.618320] LustreError: 93979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2655.043631] Lustre: server umount lustre-MDT0001 complete [ 2669.425213] Lustre: server umount lustre-OST0000 complete [ 2683.938149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2684.394190] LustreError: 98294:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2684.640912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2689.110245] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 2695.147410] LustreError: 98295:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2695.195160] LustreError: 98295:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2698.558964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2699.529979] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2704.900104] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 2714.271528] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2721.754363] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2737.597702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2737.850386] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2737.862598] LustreError: 98284:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2737.905929] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2737.924197] LustreError: 100381:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -98 [ 2737.935133] LustreError: 100381:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2737.948710] LustreError: 100381:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2738.029569] Lustre: server umount lustre-OST0000 complete [ 2738.032369] LustreError: 100381:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2748.502426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2748.810933] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2748.817896] LustreError: 98282:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2748.844416] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2748.851619] LustreError: 100845:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -98 [ 2748.858821] LustreError: 100845:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2748.863407] LustreError: 100845:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2748.926168] Lustre: server umount lustre-OST0000 complete [ 2748.928247] LustreError: 100845:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2760.349330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2760.537259] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2760.541748] Lustre: Found index 0 for lustre-OST0000, updating log [ 2760.546460] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2760.609257] Lustre: lustre-OST0000: new disk, initializing [ 2760.612223] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2760.622946] Lustre: 101380:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2760.919981] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2768.687743] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 2773.106747] Lustre: Failing over lustre-OST0000 [ 2773.378244] Lustre: server umount lustre-OST0000 complete [ 2777.489663] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2777.500457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2777.519226] Lustre: Skipped 1 previous similar message [ 2782.688888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2782.699132] Lustre: Skipped 2 previous similar messages [ 2783.955471] Lustre: server umount lustre-MDT0000 complete [ 2787.817737] LustreError: 98294:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2787.847251] LustreError: 98294:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2789.088266] LustreError: 98276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440312 with bad export cookie 14103138549404229979 [ 2789.089957] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2789.101040] LustreError: 98276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2798.835227] Lustre: server umount lustre-MDT0001 complete [ 2810.732414] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 08:58:52 (1788440332) [ 2828.963554] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 2841.161421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2841.586565] LustreError: 103734:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2841.750508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2846.567611] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 2859.324424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2865.859892] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 2869.173251] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2876.828357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2877.731664] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2877.735018] Lustre: Skipped 1 previous similar message [ 2877.745221] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2884.900843] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2884.937270] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2884.938065] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2885.344763] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 2896.211419] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2896.268434] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2896.340614] Lustre: lustre-OST0001: new disk, initializing [ 2896.343910] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2896.349187] Lustre: 106056:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2902.333724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2902.360502] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2902.567836] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2904.317354] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 2912.924566] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2924.634630] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2932.195550] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2932.210698] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2932.219075] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2936.040939] Lustre: server umount lustre-MDT0001 complete [ 2938.117778] LustreError: 106536:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2938.145414] LustreError: 106536:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2947.065950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2947.278322] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2947.290482] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2947.404385] Lustre: 103723:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2948.268331] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2948.272926] Lustre: Skipped 1 previous similar message [ 2948.714990] LustreError: 107608:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2948.720762] LustreError: 107608:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2948.725929] LustreError: 103726:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2948.871631] LustreError: 107614:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.156@tcp: cfg command failed: rc = -17 [ 2948.885575] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2948.885575] [ 2953.377944] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 2953.724072] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2966.029791] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 09:01:27 (1788440487) [ 2986.056714] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 3007.856471] Lustre: 109686:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3035.526473] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3039.806693] Lustre: 110820:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3138.022305] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3138.038314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3138.053202] Lustre: Skipped 3 previous similar messages [ 3138.060357] Lustre: Skipped 4 previous similar messages [ 3143.723935] Lustre: server umount lustre-MDT0000 complete [ 3148.260552] LustreError: 103730:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3148.290304] LustreError: 103730:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 3148.422466] LustreError: 103715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440672 with bad export cookie 14103138549404230980 [ 3148.433174] LustreError: 103715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3148.433442] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3148.803550] Lustre: server umount lustre-MDT0001 complete [ 3163.990904] Lustre: server umount lustre-OST0000 complete [ 3177.570888] Lustre: server umount lustre-OST0001 complete [ 3183.743363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3194.631855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3206.313825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3217.244945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3242.384713] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 3255.340454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3255.361330] Lustre: lustre-MDT0000: reset Object Index mappings [ 3256.046230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3261.329793] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 3271.296181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3271.320587] Lustre: lustre-MDT0001: reset Object Index mappings [ 3278.042709] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 3281.656312] Lustre: 115940:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3289.772832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3296.663437] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:227 to 0x280000400:257) [ 3297.517384] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 3300.719855] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 3307.291877] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3313.162367] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:161 to 0x2c0000401:193) [ 3313.164592] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3316.296705] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 3325.764662] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3329.870873] Lustre: 117818:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3386.801919] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 09:08:28 (1788440908) [ 3415.011860] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3415.018948] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3415.035594] Lustre: Skipped 2 previous similar messages [ 3415.045629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3415.056751] Lustre: Skipped 7 previous similar messages [ 3417.846724] Lustre: server umount lustre-MDT0000 complete [ 3420.653480] LustreError: 114787:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3420.677943] LustreError: 114787:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 3422.164469] LustreError: 114772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440946 with bad export cookie 14103138549404277852 [ 3422.185205] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3422.189538] LustreError: 114772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3425.779358] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3425.810358] Lustre: Skipped 3 previous similar messages [ 3428.628313] Lustre: server umount lustre-MDT0001 complete [ 3447.268891] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3447.456754] Lustre: server umount lustre-OST0000 complete [ 3458.430827] Lustre: server umount lustre-OST0001 complete [ 3466.643791] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [ 3481.726327] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 3523.705519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3523.927561] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3523.960443] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3524.048924] Lustre: lustre-MDT0000: new disk, initializing [ 3524.144324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3524.153281] Lustre: Skipped 3 previous similar messages [ 3524.171424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3529.283835] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 3536.096516] Lustre: server umount lustre-MDT0000 complete [ 3545.801232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3546.086564] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3546.127351] Lustre: 122117:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3546.149821] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3550.992880] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 3555.940826] Lustre: server umount lustre-MDT0000 complete [ 3565.552393] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 09:11:26 (1788441086) [ 3567.483896] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3569.522908] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 09:11:31 (1788441091) [ 3590.410073] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 3602.196813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3607.630209] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 3617.291824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3617.515356] Lustre: 124169:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3617.640787] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3617.654748] Lustre: Skipped 1 previous similar message [ 3617.724169] Lustre: lustre-MDT0001: new disk, initializing [ 3617.973782] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3617.986523] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3623.233973] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 3628.818317] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3636.107457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3636.638457] Lustre: lustre-OST0000: new disk, initializing [ 3636.646432] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3636.661769] Lustre: 125745:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3638.765953] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3638.774766] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3638.976805] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3644.076526] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 3655.687259] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3655.853767] Lustre: lustre-OST0001: new disk, initializing [ 3655.858358] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3655.864674] Lustre: 126616:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3657.084038] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3657.090591] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3657.184915] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3662.750716] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 3672.630537] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3676.565692] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4128.007688] Lustre: 133041:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4128.020358] Lustre: 133041:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4128.030718] Lustre: 133041:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4128.041198] Lustre: 133041:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4128.057653] Lustre: 133041:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4128.066301] Lustre: 133041:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4457.161580] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 09:26:18 (1788441978) [ 4474.342769] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4474.359106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4474.364939] Lustre: Skipped 3 previous similar messages [ 4474.386334] Lustre: Skipped 10 previous similar messages [ 4479.457170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4479.728737] Lustre: server umount lustre-MDT0000 complete [ 4483.616118] LustreError: 133067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442007 with bad export cookie 14103138549404347446 [ 4483.625394] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4483.637529] LustreError: 133067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4484.576523] LustreError: 124178:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4484.579479] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4484.598354] LustreError: 124178:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4484.632919] Lustre: Skipped 1 previous similar message [ 4484.646563] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4484.656471] Lustre: Skipped 4 previous similar messages [ 4490.220138] Lustre: server umount lustre-MDT0001 complete [ 4500.522971] Lustre: server umount lustre-OST0000 complete [ 4519.391586] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4519.699866] Lustre: server umount lustre-OST0001 complete [ 4528.549871] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [ 4541.462538] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 4582.214993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4582.500918] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4582.543184] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4582.702807] Lustre: lustre-MDT0000: new disk, initializing [ 4582.814925] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4582.824987] Lustre: Skipped 5 previous similar messages [ 4582.841598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4588.309173] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 4600.502314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4600.712068] Lustre: 136115:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4600.818370] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4600.824948] Lustre: Skipped 1 previous similar message [ 4600.963248] Lustre: lustre-MDT0001: new disk, initializing [ 4601.108545] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4601.136188] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4607.428798] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 4618.930993] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4624.779650] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4632.288282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4632.574860] Lustre: lustre-OST0000: new disk, initializing [ 4632.578617] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4632.589175] Lustre: 138084:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4634.062103] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4634.089820] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4634.234829] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4640.696480] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 4651.689883] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4663.882416] Lustre: lustre-MDD0000: changelog on [ 4666.949711] Lustre: lustre-MDD0001: changelog on [ 6893.209961] Lustre: lustre-MDD0001: changelog off [ 6895.624674] Lustre: 141621:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6896.557530] Lustre: lustre-MDD0000: changelog off [ 6906.348619] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6906.355717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6906.377649] Lustre: Skipped 2 previous similar messages [ 6906.390478] Lustre: Skipped 3 previous similar messages [ 6910.948833] Lustre: server umount lustre-OST0000 complete [ 6915.554339] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6915.563893] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6915.595471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6916.577932] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6921.313508] Lustre: server umount lustre-MDT0000 complete [ 6921.699778] LustreError: 139541:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6921.731248] LustreError: 139541:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6925.847184] LustreError: 136108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444449 with bad export cookie 14103138549405172879 [ 6925.863336] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6925.869683] LustreError: 136108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6926.479916] Lustre: server umount lustre-MDT0001 complete [ 6940.787066] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [ 6943.497025] Key type lgssc unregistered [ 6943.889367] LNet: 147277:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6943.913678] LNetError: 147277:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6945.002767] LNet: Removed LNI 192.168.204.156@tcp [ 6945.711147] Key type .llcrypt unregistered [ 6945.713216] Key type ._llcrypt unregistered [ 6967.419759] Key type ._llcrypt registered [ 6967.422260] Key type .llcrypt registered [ 6967.581909] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [ 6987.364713] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 6987.960313] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6987.969299] alg: No test for adler32 (adler32-zlib) [ 6989.047506] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 6989.227209] LNet: Added LNI 192.168.204.156@tcp [8/256/0/180] [ 6990.879174] Key type lgssc registered [ 6992.297281] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7036.448428] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7036.502540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7037.806333] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7037.856635] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7037.915313] Lustre: lustre-MDT0000: new disk, initializing [ 7038.062893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7038.082820] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7043.371109] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 7055.982709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7056.229368] Lustre: 150476:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7056.355802] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7056.364875] Lustre: Skipped 1 previous similar message [ 7056.496701] Lustre: lustre-MDT0001: new disk, initializing [ 7056.607568] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7056.680544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7056.703694] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7062.760897] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 7074.042277] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7081.081437] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7088.762030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7088.979522] Lustre: lustre-OST0000: new disk, initializing [ 7088.983684] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7088.989491] Lustre: 152449:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7089.066548] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7094.860190] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7094.871108] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7094.992414] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7096.205214] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 7106.929772] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7113.399930] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_module obdecho/obdecho [ 7115.833711] LustreError: 153452:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 7115.841270] LustreError: 153452:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 7117.981235] LustreError: 153549:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 7133.664807] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7133.677425] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7133.702362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7136.238026] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7136.241289] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7136.279904] Lustre: Skipped 2 previous similar messages [ 7137.943740] Lustre: server umount lustre-MDT0000 complete [ 7142.117633] LustreError: 150470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444665 with bad export cookie 16806203854371761525 [ 7142.121200] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7142.126081] LustreError: 150470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7142.366656] Lustre: server umount lustre-MDT0001 complete [ 7157.625951] Lustre: server umount lustre-OST0000 complete [ 7182.578955] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [ 7194.629945] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 7234.911625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7235.160084] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7235.190495] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7235.278787] Lustre: lustre-MDT0000: new disk, initializing [ 7235.361468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7235.379718] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7240.845483] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 7252.520526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7252.682815] Lustre: 157267:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7252.765713] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7252.772036] Lustre: Skipped 1 previous similar message [ 7252.870945] Lustre: lustre-MDT0001: new disk, initializing [ 7252.994288] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7253.029468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7253.042448] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7259.509705] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 7270.190629] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7276.213514] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7283.824109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7284.180222] Lustre: lustre-OST0000: new disk, initializing [ 7284.185863] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7284.192695] Lustre: 159234:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7284.310430] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7291.320630] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 7293.494975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7293.523491] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7293.610913] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7302.212433] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7307.529776] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7307.755249] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7313.826524] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7314.052413] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7319.521829] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7319.524936] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7319.552181] Lustre: Skipped 1 previous similar message [ 7319.569955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7323.072616] Lustre: server umount lustre-OST0000 complete [ 7329.761225] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7329.764370] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7329.770804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7329.780270] Lustre: Skipped 2 previous similar messages [ 7329.783332] Lustre: Skipped 1 previous similar message [ 7334.373078] Lustre: server umount lustre-MDT0000 complete [ 7334.881869] LustreError: 157278:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7334.896874] LustreError: 157278:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 7338.188509] LustreError: 157259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444862 with bad export cookie 16806203854371762484 [ 7338.196468] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7338.203239] LustreError: 157259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7338.702368] Lustre: server umount lustre-MDT0001 complete [ 7342.872428] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 10:14:24 (1788444864) [ 7344.423798] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7347.064455] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 10:14:28 (1788444868) [ 7356.756905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7357.199257] LustreError: 161648:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7357.291632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7362.536338] LustreError: 161649:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7362.700024] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 7367.649258] LustreError: 161648:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7372.622359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7372.770837] LustreError: 161649:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7378.405814] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 7388.824355] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7396.278931] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7405.308737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7405.639493] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7405.646270] Lustre: Skipped 1 previous similar message [ 7415.876653] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 7425.488972] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7441.693659] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.56@tcp (stopping) [ 7441.705463] Lustre: Skipped 1 previous similar message [ 7443.441227] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7443.470724] LustreError: 163172:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7443.497194] LustreError: 163172:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7443.840752] Lustre: server umount lustre-MDT0000 complete [ 7448.977901] LustreError: 161629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444972 with bad export cookie 16806203854371763366 [ 7449.020077] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7451.976844] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.56@tcp (stopping) [ 7453.667858] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7453.691038] Lustre: Skipped 2 previous similar messages [ 7459.813562] LustreError: 161645:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7459.837463] LustreError: 161645:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 7463.391396] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7463.596703] Lustre: server umount lustre-MDT0001 complete [ 7470.437048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7484.858869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7485.097239] LustreError: 165516:0:(llog.c:1655:llog_backup()) MGC192.168.204.156@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7485.104758] Lustre: 165516:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.156@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7492.860669] LustreError: 165522:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7492.911135] LustreError: 165522:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 7545.951360] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7545.975371] Lustre: 165516:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.156@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7558.398840] LustreError: 165523:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7558.426046] LustreError: 165523:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 48 previous similar messages [ 7586.806317] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7591.924332] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7592.442346] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 7602.890364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7620.587430] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7626.218359] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7626.349869] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 7634.720431] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7636.810743] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7643.038089] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7644.705858] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7667.170947] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7667.177134] Lustre: Skipped 1 previous similar message [ 7667.177595] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7667.184135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7667.184144] Lustre: Skipped 6 previous similar messages [ 7668.326765] Lustre: server umount lustre-MDT0000 complete [ 7673.899547] LustreError: 161629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445197 with bad export cookie 16806203854371769855 [ 7673.904479] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7673.915359] LustreError: 161629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7674.366639] Lustre: server umount lustre-MDT0001 complete [ 7689.014823] Lustre: server umount lustre-OST0000 complete [ 7715.380610] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [ 7728.317278] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 7768.596193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7768.752817] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7768.773273] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7768.835374] Lustre: lustre-MDT0000: new disk, initializing [ 7768.890778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7768.896261] Lustre: Skipped 1 previous similar message [ 7768.909789] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7773.039247] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 7785.850453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7785.957439] Lustre: 170627:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7786.183753] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7791.693656] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 7802.081843] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7809.176416] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7816.819379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7817.262422] Lustre: lustre-OST0000: new disk, initializing [ 7817.268152] Lustre: Skipped 1 previous similar message [ 7817.276334] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7817.280978] Lustre: Skipped 2 previous similar messages [ 7817.286727] Lustre: 172593:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7822.927393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7822.934996] Lustre: Skipped 1 previous similar message [ 7822.942619] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7823.079200] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7824.844523] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 7834.813271] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7840.359708] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7840.582113] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7845.816303] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7846.143252] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7848.938077] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7848.943703] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7848.959242] LustreError: Skipped 1 previous similar message [ 7848.965082] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7848.987353] Lustre: Skipped 2 previous similar messages [ 7849.007946] Lustre: Skipped 3 previous similar messages [ 7854.797792] Lustre: server umount lustre-OST0000 complete [ 7866.745670] Lustre: server umount lustre-MDT0000 complete [ 7869.413925] LustreError: 170637:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7869.431805] LustreError: 170637:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 50 previous similar messages [ 7872.095532] LustreError: 170619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445395 with bad export cookie 16806203854371786655 [ 7872.105549] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7872.113402] LustreError: 170619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7872.600434] Lustre: server umount lustre-MDT0001 complete [ 7884.222812] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 10:23:25 (1788445405) [ 7892.234399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7899.010209] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 7907.954416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7914.224851] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 7924.724409] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7934.941582] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7945.157372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7953.811872] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 7962.443270] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7975.438937] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 10:24:56 (1788445496) [ 7983.593722] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7983.595116] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7983.620159] Lustre: Skipped 2 previous similar messages [ 7983.627861] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7983.627870] Lustre: Skipped 5 previous similar messages [ 7983.658403] LustreError: Skipped 1 previous similar message [ 7987.158229] Lustre: server umount lustre-OST0000 complete [ 7997.699705] Lustre: server umount lustre-MDT0000 complete [ 8002.073494] LustreError: 174989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445525 with bad export cookie 16806203854371787537 [ 8002.089138] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8002.090156] LustreError: 174989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8002.682815] Lustre: server umount lustre-MDT0001 complete [ 8015.276486] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [ 8018.930291] Key type lgssc unregistered [ 8019.367115] LNet: 178742:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8019.372404] LNetError: 178742:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8019.389308] LNet: Removed LNI 192.168.204.156@tcp [ 8020.512986] Key type .llcrypt unregistered [ 8020.515404] Key type ._llcrypt unregistered [ 8042.631439] Key type ._llcrypt registered [ 8042.636896] Key type .llcrypt registered [ 8042.808963] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 8044.195568] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8044.318918] alg: No test for adler32 (adler32-zlib) [ 8045.559251] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8045.891658] LNet: Added LNI 192.168.204.156@tcp [8/256/0/180] [ 8047.560253] Key type lgssc registered [ 8048.889335] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8057.502676] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8057.549386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8073.184442] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8078.303357] LustreError: 179983:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.156@tcp: failed processing log, type 4: rc = -110 [ 8083.423813] LustreError: 179983:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8083.447597] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 8083.456145] LustreError: MGC192.168.204.156@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 8083.478759] LustreError: 179983:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 8083.501357] LustreError: 179983:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -22 [ 8083.514798] LustreError: 179983:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 8083.639988] Lustre: server umount lustre-OST0000 complete [ 8083.650357] LustreError: 179983:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8092.494542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8094.063612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8099.522781] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 8103.400921] LustreError: 180483:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8103.444196] LustreError: 180483:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8108.524889] LustreError: 180482:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8110.476971] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8110.970608] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8117.364149] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 8126.959875] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8133.750397] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8142.644909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8142.872238] LustreError: 182298:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8142.890903] Lustre: 182298:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.204.156@tcp: invalid local config log lustre-OST0000: rc = -22 [ 8143.126042] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8151.194877] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 8161.729059] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8167.392719] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8167.408341] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8167.433474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8168.929813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8168.940327] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8168.969343] Lustre: Skipped 1 previous similar message [ 8171.216895] Lustre: server umount lustre-MDT0000 complete [ 8174.073229] LustreError: 182805:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8174.094261] LustreError: 182805:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8175.700906] LustreError: 180464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445699 with bad export cookie 8740487782024622199 [ 8175.713067] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8175.724614] LustreError: 180464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8176.089718] Lustre: server umount lustre-MDT0001 complete [ 8191.141278] Lustre: server umount lustre-OST0000 complete [ 8202.363875] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [ 8217.734791] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 8262.260373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8262.629493] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8262.680364] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8262.823625] Lustre: lustre-MDT0000: new disk, initializing [ 8262.951167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8263.022687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8268.567471] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 8280.220373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8280.339673] Lustre: 185843:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8280.384909] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8280.388403] Lustre: Skipped 1 previous similar message [ 8280.472826] Lustre: lustre-MDT0001: new disk, initializing [ 8280.635099] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8280.686082] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8280.696603] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8286.006167] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 8297.039323] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8304.392418] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8313.504995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8313.912485] Lustre: lustre-OST0000: new disk, initializing [ 8313.920344] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8313.932904] Lustre: 187813:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8314.081103] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8321.090426] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8321.099043] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8321.277805] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8322.570078] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 8335.076393] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8340.853391] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8341.206193] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8346.696543] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8346.970800] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8350.182858] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8350.194676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8350.199387] Lustre: Skipped 3 previous similar messages [ 8355.319152] Lustre: server umount lustre-OST0000 complete [ 8360.417391] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8360.418904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8360.428888] Lustre: Skipped 2 previous similar messages [ 8360.448277] Lustre: Skipped 1 previous similar message [ 8365.544983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8365.559847] Lustre: Skipped 1 previous similar message [ 8365.967614] Lustre: server umount lustre-MDT0000 complete [ 8370.327602] LustreError: 185837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445894 with bad export cookie 8740487782024622920 [ 8370.331036] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8370.337794] LustreError: 185837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8370.687807] Lustre: server umount lustre-MDT0001 complete [ 8381.749653] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 10:31:43 (1788445903) [ 8400.313525] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [ 8401.981587] Key type lgssc unregistered [ 8402.200573] LNet: 190761:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8402.205740] LNetError: 190761:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8403.247948] LNet: Removed LNI 192.168.204.156@tcp [ 8404.197615] Key type .llcrypt unregistered [ 8404.199822] Key type ._llcrypt unregistered [ 8428.011266] Key type ._llcrypt registered [ 8428.012882] Key type .llcrypt registered [ 8428.256975] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 8430.128708] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8430.340672] alg: No test for adler32 (adler32-zlib) [ 8431.595890] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8432.028309] LNet: Added LNI 192.168.204.156@tcp [8/256/0/180] [ 8433.687796] Key type lgssc registered [ 8435.267129] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8444.065604] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8444.084952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8445.482140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8449.810497] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 8455.650064] LustreError: 191979:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8455.662587] LustreError: 191979:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8458.973932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8459.365185] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8464.099705] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 8472.149288] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8478.312960] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8486.361682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8486.525808] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8486.536463] LustreError: 193843:0:(llog.c:1696:llog_backup()) MGC192.168.204.156@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8486.550560] Lustre: 193843:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.204.156@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8486.687616] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8492.842730] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 8501.396798] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8504.808952] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8504.813504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8504.831567] Lustre: Skipped 2 previous similar messages [ 8504.842171] Lustre: Skipped 1 previous similar message [ 8509.921359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8509.935616] Lustre: Skipped 3 previous similar messages [ 8515.047097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8515.053132] Lustre: Skipped 2 previous similar messages [ 8518.111140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8518.331785] Lustre: server umount lustre-MDT0000 complete [ 8520.165573] LustreError: 194607:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8520.187861] LustreError: 194607:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8522.477150] LustreError: 191961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446046 with bad export cookie 4867924919499439254 [ 8522.487611] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8522.494521] LustreError: 191961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8523.295506] LustreError: 192709:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000005e3d4215 x1875321505740544/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8523.402181] Lustre: server umount lustre-MDT0001 complete [ 8538.351065] Lustre: server umount lustre-OST0000 complete [ 8551.229882] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [ 8565.131283] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 8606.570104] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8607.090789] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8607.143784] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8607.336931] Lustre: lustre-MDT0000: new disk, initializing [ 8607.567821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8607.587921] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8612.769443] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 8623.788807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8623.939307] Lustre: 197373:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8624.036704] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8624.041645] Lustre: Skipped 1 previous similar message [ 8624.046380] LustreError: 198157:0:(llog.c:1696:llog_backup()) MGC192.168.204.156@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8624.061041] LustreError: 198157:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8624.071382] Lustre: 198157:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.156@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8624.122845] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8624.138479] Lustre: Skipped 1 previous similar message [ 8624.262554] Lustre: lustre-MDT0001: new disk, initializing [ 8624.371710] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8624.419790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8624.438707] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8629.330736] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 8638.766496] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8644.505318] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8652.411765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8652.810620] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8652.815697] Lustre: Skipped 1 previous similar message [ 8652.820586] LustreError: 199318:0:(llog.c:1696:llog_backup()) MGC192.168.204.156@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8652.836946] LustreError: 199318:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8652.846407] Lustre: 199318:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.156@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8652.858563] Lustre: 199318:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8652.877848] Lustre: lustre-OST0000: new disk, initializing [ 8652.884688] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8652.891919] Lustre: 199343:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8652.992570] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8659.014411] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8659.025288] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8659.141798] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8660.961720] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 8670.531973] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8675.804892] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8676.007454] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8680.509356] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8680.727389] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8685.029135] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8685.030861] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8685.034350] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8685.050087] Lustre: Skipped 1 previous similar message [ 8685.067631] LustreError: Skipped 1 previous similar message [ 8690.150351] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8690.157217] Lustre: Skipped 2 previous similar messages [ 8697.311146] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8697.507526] Lustre: server umount lustre-OST0000 complete [ 8705.508708] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8705.509973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8705.534480] Lustre: Skipped 1 previous similar message [ 8705.543950] Lustre: Skipped 3 previous similar messages [ 8716.255122] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8716.484424] Lustre: server umount lustre-MDT0000 complete [ 8720.573048] LustreError: 199342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446244 with bad export cookie 4867924919499439968 [ 8720.575060] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8720.588133] LustreError: 199342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8720.861753] Lustre: server umount lustre-MDT0001 complete [ 8730.327954] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 10:37:32 (1788446252) [ 8747.638329] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 8759.026450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8759.279468] LustreError: 202476:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8759.338295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8764.385393] LustreError: 202477:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8764.463399] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 8769.510975] LustreError: 202476:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8773.650587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8779.048141] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 8782.086231] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8788.946731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8796.016349] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 8804.331455] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8804.458395] Lustre: lustre-OST0001: new disk, initializing [ 8804.461178] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8804.465580] Lustre: 204784:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8804.536704] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8804.542962] Lustre: Skipped 2 previous similar messages [ 8810.121605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8810.127984] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8810.246558] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8811.184247] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 8822.270659] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8828.410320] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8830.532871] Lustre: DEBUG MARKER: ADD OST3 [ 8836.081120] loop: module loaded [ 8836.161593] loop0: detected capacity change from 0 to 204800000 [ 8836.437348] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8836.451775] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8836.464725] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8842.964205] loop0: detected capacity change from 0 to 204800000 [ 8843.611305] Lustre: DEBUG MARKER: STOP OST3 [ 8849.188276] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8849.467797] Lustre: lustre-OST0003: new disk, initializing [ 8849.472977] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8849.485092] Lustre: 206809:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8851.492922] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8851.611608] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8851.621176] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8851.656685] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8852.973787] Lustre: Failing over lustre-OST0003 [ 8853.047779] Lustre: server umount lustre-OST0003 complete [ 8853.535076] LustreError: 206968:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8856.409690] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 8856.858341] LustreError: 203958:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.204.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8857.070971] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8857.071111] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8857.096700] Lustre: Skipped 1 previous similar message [ 8866.300335] Lustre: DEBUG MARKER: START OST3 again [ 8874.501386] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8874.813296] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8874.824958] Lustre: Skipped 1 previous similar message [ 8874.839474] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8876.518471] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8876.840386] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8876.844465] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8881.537504] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 8902.106160] Lustre: Failing over lustre-OST0003 [ 8902.260113] Lustre: server umount lustre-OST0003 complete [ 8902.627528] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8902.628277] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8902.638252] LustreError: Skipped 1 previous similar message [ 8902.652637] LustreError: 203958:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8902.652647] LustreError: 203958:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 8921.074237] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8921.075173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8921.088479] Lustre: Skipped 4 previous similar messages [ 8921.099278] Lustre: Skipped 5 previous similar messages [ 8925.502495] Lustre: server umount lustre-MDT0000 complete [ 8929.271972] LustreError: 202458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446453 with bad export cookie 4867924919499440850 [ 8929.275894] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8929.281396] LustreError: 202458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8935.852794] Lustre: server umount lustre-MDT0001 complete [ 8940.367716] Lustre: server umount lustre-OST0000 complete [ 8944.441471] Lustre: server umount lustre-OST0001 complete [ 8952.484873] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [ 8961.143169] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 8998.243620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8998.503260] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8998.550251] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8998.642317] Lustre: lustre-MDT0000: new disk, initializing [ 8998.742671] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9003.138623] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 9014.557367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9014.698501] Lustre: 212245:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9014.773695] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9014.781486] Lustre: Skipped 1 previous similar message [ 9014.895290] Lustre: lustre-MDT0001: new disk, initializing [ 9015.011868] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9015.021675] Lustre: Skipped 1 previous similar message [ 9015.054408] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9015.071769] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9020.185209] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 9030.073697] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9036.097668] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9043.630923] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9043.892367] Lustre: 214216:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9045.833987] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9045.962523] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9052.026065] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 9061.636826] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9067.397148] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9067.657741] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9071.839052] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9072.026224] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9076.708145] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9076.726053] LustreError: Skipped 1 previous similar message [ 9076.726164] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9076.740041] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9076.751056] Lustre: Skipped 3 previous similar messages [ 9076.772324] Lustre: Skipped 5 previous similar messages [ 9080.044975] Lustre: server umount lustre-OST0000 complete [ 9086.951803] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9089.864294] Lustre: server umount lustre-MDT0000 complete [ 9092.066445] LustreError: 212252:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9092.080595] LustreError: 212252:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 18 previous similar messages [ 9092.923719] LustreError: 212237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446616 with bad export cookie 4867924919499443153 [ 9092.929255] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9092.932784] LustreError: 212237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9093.345774] Lustre: server umount lustre-MDT0001 complete [ 9096.433369] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 10:43:38 (1788446618) [ 9111.548449] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [ 9120.869982] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 9156.620125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9156.906140] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9156.939990] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9156.953326] Lustre: Skipped 1 previous similar message [ 9157.060385] Lustre: lustre-MDT0000: new disk, initializing [ 9157.066841] Lustre: Skipped 1 previous similar message [ 9157.144913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9157.150733] Lustre: Skipped 1 previous similar message [ 9162.200949] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 9173.913799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9174.130633] Lustre: 218821:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9174.408146] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9179.264410] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 9190.095124] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9197.032195] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9204.114115] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9204.467659] Lustre: 220796:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9206.067510] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9210.898305] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 9220.361623] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9225.596965] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9225.827989] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9230.550410] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9230.770116] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9236.960361] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9236.963183] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9236.978307] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9236.995639] Lustre: Skipped 3 previous similar messages [ 9237.022487] Lustre: Skipped 3 previous similar messages [ 9247.206033] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9247.428450] Lustre: server umount lustre-OST0000 complete [ 9257.406534] Lustre: server umount lustre-MDT0000 complete [ 9257.446073] LustreError: 219704:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9257.466836] LustreError: 219704:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9261.361890] LustreError: 218815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446785 with bad export cookie 4867924919499444035 [ 9261.368270] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9261.372819] LustreError: 218815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9261.865022] Lustre: server umount lustre-MDT0001 complete [ 9270.664233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9271.167634] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9271.174189] Lustre: Skipped 4 previous similar messages [ 9275.462885] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 9285.172424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9290.702312] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 9297.847895] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9303.282940] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9310.435621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9316.962526] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 9324.225367] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9335.780230] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9335.803261] Lustre: Skipped 3 previous similar messages [ 9336.725184] Lustre: server umount lustre-OST0000 complete [ 9346.902901] Lustre: server umount lustre-MDT0000 complete [ 9350.046061] LustreError: 223749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446873 with bad export cookie 4867924919499444917 [ 9350.055930] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9350.057094] LustreError: 223749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9350.442452] Lustre: server umount lustre-MDT0001 complete [ 9461.676274] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [ 9464.086380] Key type lgssc unregistered [ 9464.309594] LNet: 226675:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9464.319502] LNetError: 226675:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9465.382296] LNet: Removed LNI 192.168.204.156@tcp [ 9466.222394] Key type .llcrypt unregistered [ 9466.232637] Key type ._llcrypt unregistered [ 9489.207367] Key type ._llcrypt registered [ 9489.209688] Key type .llcrypt registered [ 9489.316814] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [ 9505.073576] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 9505.796854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9505.828685] alg: No test for adler32 (adler32-zlib) [ 9506.952086] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 9507.199391] LNet: Added LNI 192.168.204.156@tcp [8/256/0/180] [ 9508.879190] Key type lgssc registered [ 9510.053640] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9543.444747] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9543.477404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9544.689408] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9544.743574] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9544.796630] Lustre: lustre-MDT0000: new disk, initializing [ 9544.845253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9544.858626] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9548.990872] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 9558.717402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9558.831729] Lustre: 230025:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9558.894995] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9558.901344] Lustre: Skipped 1 previous similar message [ 9558.994894] Lustre: lustre-MDT0001: new disk, initializing [ 9559.073603] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9559.130812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9559.139786] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9563.651709] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 9572.458270] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9577.979156] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9585.822259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9586.102919] Lustre: lustre-OST0000: new disk, initializing [ 9586.109475] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9586.116270] Lustre: 231993:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9586.246829] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9592.397985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9592.414664] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9592.530779] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9593.507672] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 9602.990348] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9607.636142] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9607.878296] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9612.316628] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9612.493661] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9618.403062] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9618.403346] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9618.409827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9618.428237] LustreError: Skipped 1 previous similar message [ 9618.442907] Lustre: Skipped 1 previous similar message [ 9620.609983] Lustre: server umount lustre-OST0000 complete [ 9625.575617] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9625.583401] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9625.599723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9625.607600] Lustre: Skipped 1 previous similar message [ 9628.651668] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9628.653121] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9633.760569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9633.770314] Lustre: Skipped 2 previous similar messages [ 9638.881225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9638.886799] Lustre: Skipped 1 previous similar message [ 9639.391136] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9639.589070] Lustre: server umount lustre-MDT0000 complete [ 9643.402094] LustreError: 230017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447167 with bad export cookie 18217423229881810425 [ 9643.418300] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9643.419194] LustreError: 230017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9643.828248] Lustre: server umount lustre-MDT0001 complete [ 9664.813891] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 9675.578279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9675.976268] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9681.078929] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 9681.378564] LustreError: 234930:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9681.400242] LustreError: 234930:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9685.477584] LustreError: 234931:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9690.446421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9690.604066] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9690.856439] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9695.834219] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 9699.265213] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9706.077970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9706.524171] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9712.894794] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 9721.246469] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9721.417130] Lustre: lustre-OST0001: new disk, initializing [ 9721.420458] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9721.426202] Lustre: 237236:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9726.584228] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9726.598110] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9726.680821] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9727.406982] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 9738.044775] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9744.987164] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9757.195985] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 10:54:38 (1788447278) [ 9762.792147] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9762.794891] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9762.802075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9762.806973] Lustre: Skipped 1 previous similar message [ 9765.534418] Lustre: server umount lustre-MDT0000 complete [ 9767.912532] LustreError: 234931:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9767.928904] LustreError: 234931:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9768.914632] LustreError: 234910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447292 with bad export cookie 18217423229881811314 [ 9768.926753] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9768.927437] LustreError: 234910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9769.203769] Lustre: server umount lustre-MDT0001 complete [ 9782.459790] Lustre: server umount lustre-OST0000 complete [ 9795.558344] Lustre: server umount lustre-OST0001 complete [ 9801.473351] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [ 9810.450384] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 9842.650317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9842.827561] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9842.851106] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9842.915765] Lustre: lustre-MDT0000: new disk, initializing [ 9842.994947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9843.002155] Lustre: Skipped 1 previous similar message [ 9843.035338] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9847.598649] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 9857.614940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9857.757676] Lustre: 241183:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9857.807332] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9857.813867] Lustre: Skipped 1 previous similar message [ 9857.916503] Lustre: lustre-MDT0001: new disk, initializing [ 9858.044717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9858.054891] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9861.917350] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 9871.431602] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9877.378726] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9883.541860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9883.786765] Lustre: lustre-OST0000: new disk, initializing [ 9883.790600] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9883.795600] Lustre: 243152:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9883.898857] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9883.903501] Lustre: Skipped 1 previous similar message [ 9885.573723] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9885.582953] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9885.682865] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9889.618571] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 9898.572426] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9903.607195] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9903.825466] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9908.480346] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9908.710948] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9911.264739] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9911.269755] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9911.280725] LustreError: Skipped 1 previous similar message [ 9911.281826] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9911.281832] Lustre: Skipped 3 previous similar messages [ 9911.324822] Lustre: Skipped 3 previous similar messages [ 9916.731942] Lustre: server umount lustre-OST0000 complete [ 9921.505542] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9927.811966] Lustre: server umount lustre-MDT0000 complete [ 9931.747380] LustreError: 242553:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9931.784829] LustreError: 242553:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9931.916069] LustreError: 241177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447455 with bad export cookie 18217423229881813449 [ 9931.922202] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9931.938656] LustreError: 241177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9932.273903] Lustre: server umount lustre-MDT0001 complete [ 9942.537389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9943.094734] LustreError: 245366:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9948.463480] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 9952.748440] LustreError: 245367:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9952.766925] LustreError: 245367:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9958.135654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9958.437279] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9958.444782] Lustre: Skipped 1 previous similar message [ 9963.198177] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 9972.354591] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9979.032846] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9987.276019] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9994.967566] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [10004.069856] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10152.425339] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10152.429543] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10152.432635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10152.432643] Lustre: Skipped 7 previous similar messages [10152.450519] Lustre: Skipped 2 previous similar messages [10152.504500] LustreError: Skipped 1 previous similar message [10156.448627] Lustre: server umount lustre-OST0000 complete [10167.154418] Lustre: server umount lustre-MDT0000 complete [10167.778696] LustreError: 246819:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10167.801282] LustreError: 246819:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10171.075951] LustreError: 245349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447694 with bad export cookie 18217423229881814331 [10171.087768] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10171.091355] LustreError: 245349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10171.530646] Lustre: server umount lustre-MDT0001 complete [10276.796712] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [10281.349934] Key type lgssc unregistered [10281.729323] LNet: 248986:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10281.752201] LNetError: 248986:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10281.769426] LNet: Removed LNI 192.168.204.156@tcp [10283.203151] Key type .llcrypt unregistered [10283.204820] Key type ._llcrypt unregistered [10312.357385] Key type ._llcrypt registered [10312.364270] Key type .llcrypt registered [10312.509717] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [10327.912423] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [10328.767789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10328.868947] alg: No test for adler32 (adler32-zlib) [10330.048592] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [10330.354289] LNet: Added LNI 192.168.204.156@tcp [8/256/0/180] [10332.025416] Key type lgssc registered [10333.049654] Lustre: Echo OBD driver; http://www.lustre.org/ [10372.769048] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [10383.164517] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10383.210695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10384.499781] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10384.548400] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10384.671234] Lustre: lustre-MDT0000: new disk, initializing [10384.749366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10384.769784] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10389.796587] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [10400.191226] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10400.351936] Lustre: 253052:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10400.411612] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10400.415639] Lustre: Skipped 1 previous similar message [10400.541259] Lustre: lustre-MDT0001: new disk, initializing [10400.644382] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10400.693086] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10400.698702] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10405.092851] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [10410.241631] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10418.171574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10418.533233] Lustre: lustre-OST0000: new disk, initializing [10418.542092] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10418.555639] Lustre: 254685:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10418.649176] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10421.843365] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10421.864686] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10422.006882] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10425.078093] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [10435.547398] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10435.837886] Lustre: lustre-OST0001: new disk, initializing [10435.842316] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10435.866140] Lustre: 255557:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10436.054164] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10441.850858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10441.855610] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10441.932953] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10442.901695] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [10455.107958] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10466.851262] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10482.659776] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10482.676125] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10482.704066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10483.188542] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10483.199361] Lustre: Skipped 1 previous similar message [10483.204274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10483.211288] Lustre: Skipped 3 previous similar messages [10485.564398] Lustre: server umount lustre-MDT0000 complete [10489.247116] LustreError: 254695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448013 with bad export cookie 4648485845115808665 [10489.252818] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10489.253767] LustreError: 254695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10489.541696] Lustre: server umount lustre-MDT0001 complete [10503.915742] Lustre: server umount lustre-OST0000 complete [10518.146836] Lustre: server umount lustre-OST0001 complete [10530.594742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10538.923879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10546.649860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10556.263262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10573.699593] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [10584.452291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10584.473129] Lustre: lustre-MDT0000: reset Object Index mappings [10584.894896] LustreError: 260569:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10584.924516] LustreError: 260569:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10584.992254] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10590.176101] LustreError: 260570:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10591.079882] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [10594.273594] LustreError: 260569:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10599.393946] LustreError: 260570:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10600.474410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10600.490487] Lustre: lustre-MDT0001: reset Object Index mappings [10600.888097] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10605.815520] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [10608.564368] Lustre: 261711:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10615.364937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10622.207832] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [10626.992771] LustreError: 262065:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10631.099264] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10631.161764] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10631.467529] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10631.473531] Lustre: Skipped 1 previous similar message [10636.808408] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10640.888039] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [10654.049454] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10661.592673] Lustre: 263584:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10668.286951] Lustre: server umount lustre-MDT0000 complete [10672.241054] LustreError: 260551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448196 with bad export cookie 4648485845115815455 [10672.247623] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10672.268535] LustreError: 260551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10672.609583] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10672.627446] Lustre: Skipped 3 previous similar messages [10672.635091] LustreError: 261775:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10672.662183] LustreError: 261775:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10672.767713] Lustre: server umount lustre-MDT0001 complete [10678.852926] Lustre: server umount lustre-OST0000 complete [10683.020858] Lustre: server umount lustre-OST0001 complete [10703.321367] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 11:10:24 (1788448224) [10721.647205] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [10735.593233] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [10773.080027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10773.229950] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10773.300160] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10773.400768] Lustre: lustre-MDT0000: new disk, initializing [10773.551122] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10773.584000] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10778.219473] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [10790.136933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10790.279764] Lustre: 267834:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10790.389466] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10790.402061] Lustre: Skipped 1 previous similar message [10790.474850] Lustre: lustre-MDT0001: new disk, initializing [10790.579886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10790.604944] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10794.973679] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [10804.435159] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10810.182624] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10817.368694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10817.774402] Lustre: lustre-OST0000: new disk, initializing [10817.778228] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10817.783996] Lustre: 269804:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10819.277165] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10819.289968] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10819.401133] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10825.290318] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [10835.993384] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10841.073799] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10841.273599] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10845.760527] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10845.948936] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10850.276666] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10850.279836] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10850.290204] Lustre: Skipped 2 previous similar messages [10850.304624] Lustre: Skipped 1 previous similar message [10855.393878] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10860.531024] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10860.542309] Lustre: Skipped 2 previous similar messages [10862.047157] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10862.248415] Lustre: server umount lustre-OST0000 complete [10867.685162] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10867.695881] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10870.763361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10870.768650] Lustre: Skipped 2 previous similar messages [10872.535311] Lustre: server umount lustre-MDT0000 complete [10875.880571] LustreError: 267847:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10875.914052] LustreError: 267847:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10876.552406] LustreError: 268639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448400 with bad export cookie 4648485845115817548 [10876.554369] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10876.558514] LustreError: 268639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10877.036585] Lustre: server umount lustre-MDT0001 complete [10898.968782] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [10909.126750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10909.487591] LustreError: 272740:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10909.539850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10909.547715] Lustre: Skipped 2 previous similar messages [10914.335472] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [10923.813963] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10928.968923] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [10931.977426] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10939.069944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10945.697977] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [10955.517296] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10955.685979] Lustre: lustre-OST0001: new disk, initializing [10955.692655] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10955.701893] Lustre: 275049:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10957.361518] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10957.372365] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10957.488519] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10962.670615] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [10972.702904] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10976.450900] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10982.058854] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10982.078841] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10993.428898] Lustre: server umount lustre-MDT0000 complete [10993.649794] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10993.664251] Lustre: Skipped 3 previous similar messages [10993.675310] LustreError: 274522:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10993.691315] LustreError: 274522:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10998.445533] LustreError: 272721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448522 with bad export cookie 4648485845115818430 [10998.449547] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10998.462734] LustreError: 272721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10999.011967] Lustre: server umount lustre-MDT0001 complete [11004.666410] Lustre: server umount lustre-OST0000 complete [11009.751503] Lustre: server umount lustre-OST0001 complete [11030.711774] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [11043.003955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11043.702247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11043.707447] Lustre: Skipped 3 previous similar messages [11049.649283] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [11060.046958] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11065.648782] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [11068.938596] Lustre: 279532:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11076.257144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11084.000837] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [11088.249398] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [11094.401138] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11100.187744] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [11102.130207] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [11111.856943] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11116.336275] Lustre: 281412:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11121.822296] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [11122.216908] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [11136.621625] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 11:17:38 (1788448658) [11151.329116] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11151.343367] Lustre: Skipped 3 previous similar messages [11151.346025] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11151.354316] Lustre: Skipped 2 previous similar messages [11157.666541] Lustre: server umount lustre-MDT0000 complete [11161.582790] LustreError: 278389:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11161.604646] LustreError: 278389:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 16 previous similar messages [11162.059881] LustreError: 278372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448685 with bad export cookie 4648485845115824856 [11162.060948] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11162.068311] LustreError: 278372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11168.627993] Lustre: server umount lustre-MDT0001 complete [11179.510203] Lustre: server umount lustre-OST0000 complete [11183.329345] Lustre: server umount lustre-OST0001 complete [11190.311572] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [11200.972273] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [11250.458501] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [11261.353037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11261.636442] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11261.665952] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11261.736553] Lustre: lustre-MDT0000: new disk, initializing [11261.885551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11266.223547] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [11276.932328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11277.044665] Lustre: 285941:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11277.290220] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11282.322611] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [11295.641113] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11295.959393] Lustre: 287575:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11298.071761] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11302.852221] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [11312.876071] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11313.027936] Lustre: 288444:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11313.154773] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11313.166623] Lustre: Skipped 6 previous similar messages [11315.069482] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11315.075727] Lustre: Skipped 1 previous similar message [11315.168959] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11318.977574] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [11328.854857] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11332.931993] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11332.936621] Lustre: Skipped 1 previous similar message [11349.159721] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 11:21:10 (1788448870) [11351.888279] Lustre: server umount lustre-MDT0000 complete [11354.083296] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11354.093899] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11354.108913] Lustre: Skipped 4 previous similar messages [11356.649344] LustreError: 285931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448880 with bad export cookie 4648485845115832787 [11356.658164] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11356.671321] LustreError: 285931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11357.135995] Lustre: server umount lustre-MDT0001 complete [11371.480932] Lustre: server umount lustre-OST0000 complete [11386.543883] Lustre: server umount lustre-OST0001 complete [11398.925368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11403.757096] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [11411.422116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11416.774478] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [11424.961973] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11431.551673] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11439.282072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11444.730218] LustreError: 292522:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11444.760032] LustreError: 292522:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [11445.311431] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [11453.207566] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11474.033465] loop1: detected capacity change from 0 to 204800000 [11474.311530] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11474.321261] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11474.332842] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11480.114452] loop1: detected capacity change from 0 to 204800000 [11486.179429] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11486.292095] LustreError: 290681:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs27909: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11486.306509] LustreError: 290681:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs27909 [11486.311597] LustreError: 290681:0:(mgs_handler.c:517:mgs_target_reg()) fs27909-MDT0001: Error checking target index: rc = -13 [11486.331318] LustreError: fs27909-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11486.338792] LustreError: 294411:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -13 [11486.346514] LustreError: 294411:0:(tgt_mount.c:2132:server_put_super()) no obd fs27909-MDT0001 [11486.352347] LustreError: 294411:0:(tgt_mount.c:116:server_deregister_mount()) fs27909-MDT0001 not registered [11486.398474] Lustre: server umount fs27909-MDT0001 complete [11486.401396] LustreError: 294411:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11495.625653] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11495.764763] Lustre: Setting parameter fs27909-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs27909-MDT0001 [11495.819975] Lustre: srv-fs27909-MDT0001: No data found on store. Initialize space. [11495.826242] Lustre: Skipped 4 previous similar messages [11495.946573] Lustre: fs27909-MDT0001: new disk, initializing [11495.955434] Lustre: Skipped 3 previous similar messages [11496.087068] Lustre: srv-fs27909-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11496.101399] Lustre: 294878:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs27909-MDT0001-mdtlov: can't create new llog: rc = -115 [11496.116153] LustreError: 294878:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs27909-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11501.467709] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [11508.591952] Lustre: server umount fs27909-MDT0001 complete [11523.592823] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11524.158459] loop1: detected capacity change from 0 to 204800000 [11529.729840] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11530.011817] Lustre: 296446:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs27909-MDT0001-mdtlov: can't create new llog: rc = -115 [11530.024491] LustreError: 296446:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs27909-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11534.578799] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [11538.937584] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11539.812675] Lustre: server umount fs27909-MDT0001 complete [11550.176851] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11550.182575] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11550.188996] Lustre: Skipped 3 previous similar messages [11550.200096] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11550.202595] Lustre: Skipped 9 previous similar messages [11556.240135] Lustre: server umount lustre-MDT0000 complete [11566.879911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11567.024690] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11572.746576] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11573.272123] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [11582.423425] loop1: detected capacity change from 0 to 204800000 [11588.642337] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11589.360608] Lustre: 298892:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs27909-MDT0001-mdtlov: can't create new llog: rc = -115 [11589.373624] LustreError: 298892:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs27909-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11595.651691] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [11601.613212] Lustre: server umount fs27909-MDT0001 complete [11625.955065] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11625.964430] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11625.972185] Lustre: Skipped 4 previous similar messages [11639.263130] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11639.493996] Lustre: server umount lustre-OST0000 complete [11649.515137] Lustre: server umount lustre-MDT0000 complete [11653.539165] LustreError: 290673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449177 with bad export cookie 4648485845115835468 [11653.563036] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11654.033275] Lustre: server umount lustre-MDT0001 complete [11666.745948] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [11669.932051] Key type lgssc unregistered [11670.345928] LNet: 301236:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11670.358309] LNetError: 301236:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11670.384318] LNet: Removed LNI 192.168.204.156@tcp [11671.323278] Key type .llcrypt unregistered [11671.325637] Key type ._llcrypt unregistered [11692.264486] Key type ._llcrypt registered [11692.268562] Key type .llcrypt registered [11692.369691] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [11693.513178] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11693.570086] alg: No test for adler32 (adler32-zlib) [11694.798498] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [11695.059536] LNet: Added LNI 192.168.204.156@tcp [8/256/0/180] [11696.751211] Key type lgssc registered [11697.942347] Lustre: Echo OBD driver; http://www.lustre.org/ [11706.041220] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11706.056940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11707.680956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11712.588703] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [11718.118654] LustreError: 302504:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11718.141408] LustreError: 302504:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11722.255540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11722.656225] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11726.917791] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [11735.100811] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11740.573844] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11748.227777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11748.915755] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11753.041158] LustreError: 304336:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11756.348478] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [11758.060463] LustreError: 304338:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11763.184637] LustreError: 304337:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11766.549649] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11767.271621] LustreError: 304531:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11767.333598] LustreError: 304531:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11777.508991] LustreError: 304336:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11777.556096] LustreError: 304336:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11787.734635] loop1: detected capacity change from 0 to 204800000 [11787.958416] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11787.975687] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11787.984930] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11794.048870] loop1: detected capacity change from 0 to 204800000 [11795.225914] LustreError: 304336:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11795.259316] LustreError: 304336:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [11800.765607] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11800.988485] Lustre: fs10341-OST0000: new disk, initializing [11800.994587] Lustre: srv-fs10341-OST0000: No data found on store. Initialize space. [11801.017216] Lustre: 306244:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11801.150607] Lustre: fs10341-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11808.610875] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [11819.139406] Lustre: server umount fs10341-OST0000 complete [11828.713342] LustreError: 304338:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11828.742702] LustreError: 304338:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 19 previous similar messages [11832.075539] loop1: detected capacity change from 0 to 204800000 [11833.858067] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11838.310885] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11838.676863] Lustre: fs10341-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11846.284861] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [11849.212462] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11857.291391] loop2: detected capacity change from 0 to 204800000 [11857.579907] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11857.591886] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11857.611142] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11863.709135] loop2: detected capacity change from 0 to 204800000 [11870.356620] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11870.443550] LustreError: fs10341-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11870.458822] LustreError: fs10341-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11870.475811] LustreError: 309287:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -13 [11870.480329] LustreError: 309287:0:(tgt_mount.c:2132:server_put_super()) no obd fs10341-OST0001 [11870.484812] LustreError: 309287:0:(tgt_mount.c:116:server_deregister_mount()) fs10341-OST0001 not registered [11870.511896] Lustre: server umount fs10341-OST0001 complete [11870.515250] LustreError: 309287:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11882.690787] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11882.987983] Lustre: fs10341-OST0001: new disk, initializing [11882.997936] Lustre: srv-fs10341-OST0001: No data found on store. Initialize space. [11883.007936] Lustre: 309839:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11883.165764] Lustre: fs10341-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11890.236493] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [11895.273846] LustreError: 304337:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11895.293871] LustreError: 304337:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [11898.364934] Lustre: server umount fs10341-OST0000 complete [11907.804852] Lustre: server umount fs10341-OST0001 complete [11919.884204] loop1: detected capacity change from 0 to 204800000 [11925.959625] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11926.414822] Lustre: fs10341-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11932.504159] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [11940.173557] loop2: detected capacity change from 0 to 204800000 [11945.873768] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11953.653248] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [11962.805937] Lustre: server umount fs10341-OST0000 complete [11972.285782] Lustre: server umount fs10341-OST0001 complete [11987.440512] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11987.443522] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11987.457668] LustreError: Skipped 1 previous similar message [11987.461029] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11987.491479] Lustre: Skipped 1 previous similar message [11992.545741] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11992.569490] Lustre: Skipped 1 previous similar message [11993.225288] Lustre: server umount lustre-OST0000 complete [11999.208712] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11999.229270] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11999.265552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12002.785707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12002.788210] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12004.902505] Lustre: server umount lustre-MDT0000 complete [12008.225424] LustreError: 302487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449532 with bad export cookie 17036080552213054875 [12008.250539] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12008.562809] Lustre: server umount lustre-MDT0001 complete [12019.362963] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [12022.120521] Key type lgssc unregistered [12022.436968] LNet: 315760:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12022.443445] LNetError: 315760:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12022.463944] LNet: Removed LNI 192.168.204.156@tcp [12023.392425] Key type .llcrypt unregistered [12023.398714] Key type ._llcrypt unregistered [12057.596022] Key type ._llcrypt registered [12057.597792] Key type .llcrypt registered [12057.681375] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [12074.403350] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [12075.558225] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12075.581337] alg: No test for adler32 (adler32-zlib) [12076.653320] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12076.959926] LNet: Added LNI 192.168.204.156@tcp [8/256/0/180] [12078.623176] Key type lgssc registered [12079.628877] Lustre: Echo OBD driver; http://www.lustre.org/ [12118.198716] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12118.240852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12119.656793] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12119.715490] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12119.863325] Lustre: lustre-MDT0000: new disk, initializing [12119.948419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12119.967613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12124.462863] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [12139.152370] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12139.392767] Lustre: 319619:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12139.576320] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12139.588042] Lustre: Skipped 1 previous similar message [12139.683729] Lustre: lustre-MDT0001: new disk, initializing [12139.801777] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12139.868801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12139.884597] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12146.358312] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [12161.592381] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12169.806474] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12177.184357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12177.489908] Lustre: lustre-OST0000: new disk, initializing [12177.493547] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12177.499410] Lustre: 321588:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12177.563526] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12183.863383] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [12186.150044] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12186.155731] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12186.213370] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12193.491671] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12196.062737] Lustre: Mounted lustre-client - version 2.17.58_2_gee80f9b [12197.026854] Lustre: Unmounted lustre-client [12206.560819] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12206.576863] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12206.604953] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12206.613862] Lustre: Skipped 1 previous similar message [12211.702115] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12211.709488] Lustre: Skipped 1 previous similar message [12216.811286] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12216.823314] Lustre: Skipped 1 previous similar message [12220.385781] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12220.583324] Lustre: server umount lustre-OST0000 complete [12227.041692] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12227.048057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12227.055025] Lustre: Skipped 2 previous similar messages [12230.652536] Lustre: server umount lustre-MDT0000 complete [12234.406545] LustreError: 319613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449758 with bad export cookie 4030733479525254817 [12234.415544] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12234.813939] Lustre: server umount lustre-MDT0001 complete [12246.950684] LNet: 1 local NIs in recovery (showing 1): 192.168.204.156@tcp [12247.289780] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [12250.832384] Key type lgssc unregistered [12251.125695] LNet: 323753:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12251.137270] LNetError: 323753:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12325.925339] LNet: Removed LNI 192.168.204.156@tcp [12326.784422] Key type .llcrypt unregistered [12326.786831] Key type ._llcrypt unregistered [12344.321504] Key type ._llcrypt registered [12344.323646] Key type .llcrypt registered [12344.451755] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [12358.942331] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [12360.586520] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12360.628445] alg: No test for adler32 (adler32-zlib) [12361.862345] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12362.136660] LNet: Added LNI 192.168.204.156@tcp [8/256/0/180] [12363.849689] Key type lgssc registered [12365.007849] Lustre: Echo OBD driver; http://www.lustre.org/ [12395.474049] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 11:38:37 (1788449917) [12402.072242] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12402.115075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12403.440947] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12403.474665] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12403.584437] Lustre: lustre-MDT0000: new disk, initializing [12403.706509] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12403.737348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12408.649397] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [12419.166423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12419.299493] Lustre: 326969:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12419.410290] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12419.421954] Lustre: Skipped 1 previous similar message [12419.534853] Lustre: lustre-MDT0001: new disk, initializing [12419.652590] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12419.710383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12419.742767] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12424.361744] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [12434.491686] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12440.705106] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12447.180250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12447.403328] Lustre: lustre-OST0000: new disk, initializing [12447.406641] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12447.413116] Lustre: 328937:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12447.525820] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12453.812612] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [12456.504777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12456.518023] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12456.614203] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12462.886360] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12482.529435] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12482.530340] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12482.542066] LustreError: Skipped 1 previous similar message [12482.544197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12482.578037] Lustre: Skipped 1 previous similar message [12484.035435] Lustre: server umount lustre-OST0000 complete [12491.232851] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12491.241930] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12491.256941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12491.260027] Lustre: Skipped 1 previous similar message [12492.768535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12492.771863] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12496.398834] Lustre: server umount lustre-MDT0000 complete [12500.494095] LustreError: 326961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450024 with bad export cookie 6117461162651684346 [12500.499400] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12500.508682] LustreError: 326961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12500.970452] Lustre: server umount lustre-MDT0001 complete [12513.165627] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [12516.058450] Key type lgssc unregistered [12516.393515] LNet: 330845:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12516.409486] LNetError: 330845:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12516.444731] LNet: Removed LNI 192.168.204.156@tcp [12517.407780] Key type .llcrypt unregistered [12517.412721] Key type ._llcrypt unregistered [12540.992385] Key type ._llcrypt registered [12540.997036] Key type .llcrypt registered [12541.125567] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [12542.204271] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12542.343854] alg: No test for adler32 (adler32-zlib) [12543.527984] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12543.836756] LNet: Added LNI 192.168.204.156@tcp [8/256/0/180] [12545.575179] Key type lgssc registered [12546.836112] Lustre: Echo OBD driver; http://www.lustre.org/ [12554.126900] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12554.143289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12555.700810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12560.457316] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [12565.990202] LustreError: 332116:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12566.009486] LustreError: 332116:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12568.823580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12569.325720] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12574.082814] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [12583.061399] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12589.904419] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12597.908767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12598.439518] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12605.602249] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [12614.993355] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12627.717988] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12630.060969] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12635.617809] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12635.622421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12635.639664] Lustre: Skipped 1 previous similar message [12638.494043] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.56@tcp (stopping) [12638.506070] Lustre: Skipped 2 previous similar messages [12640.739969] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12640.750564] Lustre: Skipped 2 previous similar messages [12641.351387] Lustre: server umount lustre-MDT0000 complete [12643.580619] LustreError: 332112:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12645.867352] LustreError: 332111:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12645.883808] LustreError: 332111:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12645.988406] LustreError: 332097:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450169 with bad export cookie 14907544095432288798 [12645.990277] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12645.998724] LustreError: 332097:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12648.727723] LustreError: 332113:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12648.778515] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.56@tcp (stopping) [12650.976451] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12650.988564] Lustre: Skipped 1 previous similar message [12652.487602] Lustre: server umount lustre-MDT0001 complete [12657.223551] Lustre: server umount lustre-OST0000 complete [12667.357418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12672.800892] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [12676.093823] Lustre: 336299:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.156@tcp [12678.573557] Lustre: server umount MGS complete [12688.653442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12689.148868] LustreError: 336871:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12689.181052] LustreError: 336871:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12689.281488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12694.807235] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [12698.876146] LustreError: 336866:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12698.893719] LustreError: 336866:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12704.979483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12705.462616] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12710.916244] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [12720.434628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12720.845797] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12727.684984] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [12735.386128] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12737.096412] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12742.821335] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12744.725392] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12750.717148] Lustre: server umount lustre-OST0000 complete [12756.961619] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12756.971416] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12757.004820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12757.015806] Lustre: Skipped 1 previous similar message [12760.039975] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12765.157647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12765.166622] Lustre: Skipped 3 previous similar messages [12767.204213] Lustre: 331427:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788450275/real 1788450275] req@00000000553a8e45 x1875325817518080/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788450291 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12767.252952] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12770.207547] Lustre: 331426:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788450278/real 1788450278] req@00000000a6dd7102 x1875325817518976/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788450294 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12770.274104] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12770.438211] Lustre: server umount lustre-MDT0000 complete [12772.326406] LustreError: 336872:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12772.370646] LustreError: 336872:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12775.020553] LustreError: 336852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450298 with bad export cookie 14907544095432289638 [12775.034184] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12775.421538] Lustre: server umount lustre-MDT0001 complete [12788.629336] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [12791.127952] Key type lgssc unregistered [12791.394258] LNet: 340312:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12791.397824] LNetError: 340312:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12791.417290] LNet: Removed LNI 192.168.204.156@tcp [12792.387803] Key type .llcrypt unregistered [12792.390168] Key type ._llcrypt unregistered [12811.946659] Key type ._llcrypt registered [12811.952651] Key type .llcrypt registered [12812.100561] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [12826.868135] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [12827.906571] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12827.943640] alg: No test for adler32 (adler32-zlib) [12829.030048] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12829.326249] LNet: Added LNI 192.168.204.156@tcp [8/256/0/180] [12830.999248] Key type lgssc registered [12832.593070] Lustre: Echo OBD driver; http://www.lustre.org/ [12876.682964] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12876.731766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12877.938631] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12877.973147] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12878.036615] Lustre: lustre-MDT0000: new disk, initializing [12878.096391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12878.113596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12883.142129] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [12893.749990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12893.917359] Lustre: 343470:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12893.976958] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12893.983883] Lustre: Skipped 1 previous similar message [12894.074782] Lustre: lustre-MDT0001: new disk, initializing [12894.192147] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12894.239344] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12894.248842] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12898.972641] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [12907.475149] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12913.772633] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12921.022414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12921.361649] Lustre: lustre-OST0000: new disk, initializing [12921.365665] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12921.373391] Lustre: 345437:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12921.460724] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12928.144935] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [12929.101634] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12929.121725] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12929.261613] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12937.612570] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12942.769085] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12943.044348] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12947.716660] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12947.908878] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12949.999425] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12950.005453] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12950.006306] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12950.012217] Lustre: Skipped 1 previous similar message [12950.029294] LustreError: Skipped 1 previous similar message [12955.104800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12955.110606] Lustre: Skipped 2 previous similar messages [12955.935933] Lustre: server umount lustre-OST0000 complete [12960.740642] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12960.754934] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12960.772037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12965.346194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12965.355472] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12966.846211] Lustre: server umount lustre-MDT0000 complete [12972.391651] LustreError: 343461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450496 with bad export cookie 16124498829923287720 [12972.398782] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12972.411935] LustreError: 343461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12972.927747] Lustre: server umount lustre-MDT0001 complete [12982.928844] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 11:48:24 (1788450504) [13000.581232] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [13011.142959] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [13050.378066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13050.660668] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13050.685297] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13050.757385] Lustre: lustre-MDT0000: new disk, initializing [13050.831333] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13050.845914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13055.698158] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [13067.672129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13067.838709] Lustre: 350198:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13067.920918] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13067.932934] Lustre: Skipped 1 previous similar message [13068.058952] Lustre: lustre-MDT0001: new disk, initializing [13068.214988] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13068.321137] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13068.344150] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13073.287215] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [13084.724615] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13092.023652] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13099.857201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13100.077528] Lustre: lustre-OST0000: new disk, initializing [13100.081757] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13100.091297] Lustre: 352170:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13100.182624] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13107.536032] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [13108.319914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13108.329023] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13108.474747] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13117.734308] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13159.920047] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13159.921603] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13159.951629] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13159.961565] Lustre: Skipped 1 previous similar message [13163.936929] Lustre: server umount lustre-OST0000 complete [13170.148195] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13170.159493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13170.163893] Lustre: Skipped 2 previous similar messages [13170.187519] Lustre: Skipped 1 previous similar message [13174.503243] Lustre: server umount lustre-MDT0000 complete [13175.264516] LustreError: 350212:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13175.276958] LustreError: 350212:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13179.280751] LustreError: 350192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450703 with bad export cookie 16124498829923288602 [13179.282600] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13179.297233] LustreError: 350192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13179.845919] Lustre: server umount lustre-MDT0001 complete [13193.707495] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [13197.427379] Key type lgssc unregistered [13197.941740] LNet: 354343:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13197.952791] LNetError: 354343:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13197.972327] LNet: Removed LNI 192.168.204.156@tcp [13199.172366] Key type .llcrypt unregistered [13199.176398] Key type ._llcrypt unregistered [13221.696891] Key type ._llcrypt registered [13221.703464] Key type .llcrypt registered [13221.812347] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [13222.897480] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13223.049288] alg: No test for adler32 (adler32-zlib) [13224.181264] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [13224.453412] LNet: Added LNI 192.168.204.156@tcp [8/256/0/180] [13226.119411] Key type lgssc registered [13227.314514] Lustre: Echo OBD driver; http://www.lustre.org/ [13236.255665] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13236.276552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13237.703836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13242.757268] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [13247.976226] LustreError: 355616:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13248.007966] LustreError: 355616:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13252.866116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13253.098455] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [13253.273231] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13258.436513] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [13266.428884] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13273.330810] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13280.246373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13280.601698] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13286.106937] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [13292.928493] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13308.384048] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13308.400146] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13308.420903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13309.413632] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13309.425093] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13309.448513] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13309.455169] Lustre: Skipped 1 previous similar message [13312.320996] Lustre: server umount lustre-OST0000 complete [13319.654359] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13319.689897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13323.509303] Lustre: server umount lustre-MDT0000 complete [13324.799954] LustreError: 355616:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13324.822108] LustreError: 355616:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13327.764314] LustreError: 355595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450851 with bad export cookie 2064481112343470800 [13327.775317] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13327.788837] LustreError: 355595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13328.333059] Lustre: server umount lustre-MDT0001 complete [13341.267336] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [13344.729343] Key type lgssc unregistered [13345.220608] LNet: 361087:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13345.233496] LNetError: 361087:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13345.258885] LNet: Removed LNI 192.168.204.156@tcp [13346.435239] Key type .llcrypt unregistered [13346.437505] Key type ._llcrypt unregistered [13364.082786] Key type ._llcrypt registered [13364.087048] Key type .llcrypt registered [13364.224303] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [13365.282756] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13365.310650] alg: No test for adler32 (adler32-zlib) [13366.385305] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [13366.542148] LNet: Added LNI 192.168.204.156@tcp [8/256/0/180] [13368.191111] Key type lgssc registered [13369.540163] Lustre: Echo OBD driver; http://www.lustre.org/ [13378.169628] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13378.193010] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13379.727102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13384.263515] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [13389.291017] LustreError: 364771:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13389.310512] LustreError: 364771:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13393.862605] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13394.250242] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13399.481737] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [13407.704670] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13412.860795] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13420.483330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13420.842209] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13426.860716] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [13434.814029] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13449.702219] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13449.710145] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13449.726425] Lustre: Skipped 1 previous similar message [13449.731875] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13449.734749] Lustre: Skipped 1 previous similar message [13453.995203] Lustre: server umount lustre-OST0000 complete [13459.937594] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13459.940586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13459.971839] Lustre: Skipped 1 previous similar message [13459.989807] Lustre: Skipped 1 previous similar message [13465.059594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13465.070159] Lustre: Skipped 1 previous similar message [13469.181189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13469.191065] Lustre: Skipped 1 previous similar message [13474.272072] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13474.279185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13474.307153] Lustre: Skipped 1 previous similar message [13474.812496] Lustre: server umount lustre-MDT0000 complete [13479.395666] LustreError: 364765:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13479.413429] LustreError: 364765:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13479.446136] LustreError: 364692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451003 with bad export cookie 16277884325860782703 [13479.459802] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13479.464026] LustreError: 364692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13480.291966] Lustre: server umount lustre-MDT0001 complete [13495.626892] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [13499.800779] Key type lgssc unregistered [13500.211140] LNet: 372713:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13500.219583] LNetError: 372713:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13500.252489] LNet: Removed LNI 192.168.204.156@tcp [13501.448156] Key type .llcrypt unregistered [13501.449891] Key type ._llcrypt unregistered [13522.019624] Key type ._llcrypt registered [13522.021196] Key type .llcrypt registered [13522.178857] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [13523.370972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13523.507078] alg: No test for adler32 (adler32-zlib) [13524.598936] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [13524.941325] LNet: Added LNI 192.168.204.156@tcp [8/256/0/180] [13526.732369] Key type lgssc registered [13528.420625] Lustre: Echo OBD driver; http://www.lustre.org/ [13537.590481] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13537.630483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13539.185501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13544.497251] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [13548.513488] LustreError: 376477:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13548.535350] LustreError: 376477:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13553.635556] LustreError: 376476:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13554.811667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13555.350466] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13560.920538] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [13569.517692] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13576.023844] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13583.710542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13584.227912] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13591.957488] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [13600.011872] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13621.222186] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13621.234120] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13621.249386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13624.288667] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13624.303753] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13624.313263] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13624.344202] Lustre: Skipped 1 previous similar message [13626.156953] Lustre: server umount lustre-OST0000 complete [13631.968445] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13631.984791] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13632.014538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13634.528671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13634.535309] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13636.597735] Lustre: server umount lustre-MDT0000 complete [13639.658450] LustreError: 376477:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13639.689494] LustreError: 376477:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13640.447147] LustreError: 376401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451164 with bad export cookie 6904306818792376964 [13640.451564] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13640.464646] LustreError: 376401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13640.769857] Lustre: server umount lustre-MDT0001 complete [13651.910897] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [13654.731228] Key type lgssc unregistered [13655.018621] LNet: 382649:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13655.024053] LNetError: 382649:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13655.035200] LNet: Removed LNI 192.168.204.156@tcp [13655.855496] Key type .llcrypt unregistered [13655.857861] Key type ._llcrypt unregistered [13763.822770] Key type ._llcrypt registered [13763.826643] Key type .llcrypt registered [13763.934369] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [13764.310906] Key type .llcrypt unregistered [13764.312493] Key type ._llcrypt unregistered [13851.570912] Key type ._llcrypt registered [13851.585403] Key type .llcrypt registered [13851.725582] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [13852.045378] Key type .llcrypt unregistered [13852.047626] Key type ._llcrypt unregistered [13884.372099] Key type ._llcrypt registered [13884.374802] Key type .llcrypt registered [13884.502611] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [13884.846488] Key type .llcrypt unregistered [13884.850748] Key type ._llcrypt unregistered [13921.309736] Key type ._llcrypt registered [13921.314241] Key type .llcrypt registered [13921.504963] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [13921.988715] Key type .llcrypt unregistered [13921.990421] Key type ._llcrypt unregistered [14008.731490] Key type ._llcrypt registered [14008.733289] Key type .llcrypt registered [14008.863386] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [14009.236402] Key type .llcrypt unregistered [14009.238110] Key type ._llcrypt unregistered [14051.925741] Key type ._llcrypt registered [14051.927753] Key type .llcrypt registered [14052.207707] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [14052.573486] Key type .llcrypt unregistered [14052.577448] Key type ._llcrypt unregistered [14087.401536] Key type ._llcrypt registered [14087.405945] Key type .llcrypt registered [14087.539404] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [14087.858359] Key type .llcrypt unregistered [14087.863874] Key type ._llcrypt unregistered [14121.719569] Key type ._llcrypt registered [14121.722455] Key type .llcrypt registered [14121.835288] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [14122.220433] Key type .llcrypt unregistered [14122.225249] Key type ._llcrypt unregistered [14157.788554] Key type ._llcrypt registered [14157.792904] Key type .llcrypt registered [14157.964622] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [14158.343723] Key type .llcrypt unregistered [14158.349213] Key type ._llcrypt unregistered [14193.511530] Key type ._llcrypt registered [14193.518574] Key type .llcrypt registered [14193.666652] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [14193.991651] Key type .llcrypt unregistered [14193.996168] Key type ._llcrypt unregistered [14268.373983] Key type ._llcrypt registered [14268.379674] Key type .llcrypt registered [14268.538177] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [14268.838482] Key type .llcrypt unregistered [14268.841469] Key type ._llcrypt unregistered [14305.291803] Key type ._llcrypt registered [14305.297915] Key type .llcrypt registered [14305.474271] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [14305.742439] Key type .llcrypt unregistered [14305.750677] Key type ._llcrypt unregistered [14339.273219] Key type ._llcrypt registered [14339.275737] Key type .llcrypt registered [14339.394666] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [14339.693527] Key type .llcrypt unregistered [14339.695547] Key type ._llcrypt unregistered [14397.420837] Key type ._llcrypt registered [14397.425255] Key type .llcrypt registered [14397.533757] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [14416.986974] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [14418.342472] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14418.517230] alg: No test for adler32 (adler32-zlib) [14420.003371] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [14420.480725] LNet: Added LNI 192.168.204.156@tcp [8/256/0/180] [14422.231149] Key type lgssc registered [14424.214191] Lustre: Echo OBD driver; http://www.lustre.org/ [14467.482458] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14467.585674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14469.137523] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [14469.211174] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [14469.423893] Lustre: tst250fs-MDT0000: new disk, initializing [14469.604639] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14469.652501] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14477.569960] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [14491.203861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14491.347220] Lustre: 403221:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14491.403996] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [14491.409358] Lustre: Skipped 1 previous similar message [14491.483810] Lustre: tst250fs-MDT0001: new disk, initializing [14491.612085] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14491.679159] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14491.700315] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14497.437710] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [14501.983922] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [14508.483382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14508.800525] Lustre: tst250fs-OST0000: new disk, initializing [14508.804732] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [14508.810092] Lustre: 404852:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14508.957752] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14513.717083] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14513.732421] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14513.830145] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14514.734858] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [14526.431857] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14526.642273] Lustre: tst250fs-OST0001: new disk, initializing [14526.646413] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [14526.653942] Lustre: 405724:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14526.786661] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [14534.508612] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [14534.832533] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [14534.841595] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [14535.030281] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [14540.609882] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [14542.935249] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [14542.951624] Lustre: Skipped 1 previous similar message [14565.869984] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14565.888344] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14565.894179] Lustre: Skipped 2 previous similar messages [14565.909753] Lustre: Skipped 3 previous similar messages [14570.983661] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14570.989329] Lustre: Skipped 3 previous similar messages [14576.101471] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14576.117640] Lustre: Skipped 2 previous similar messages [14578.143125] Lustre: tst250fs-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14578.368663] Lustre: server umount tst250fs-MDT0000 complete [14582.799314] LustreError: 403213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452106 with bad export cookie 16301042310798945301 [14582.818001] LustreError: 403213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14582.821244] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14583.180801] Lustre: server umount tst250fs-MDT0001 complete [14596.602519] Lustre: server umount tst250fs-OST0000 complete [14610.973655] Lustre: server umount tst250fs-OST0001 complete [14632.706990] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [14643.203796] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [14684.566346] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 12:16:45 (1788452205) [14686.723967] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [14705.022981] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [14715.536562] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [14749.532607] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14486 sec ========= 12:17:50 (1788452270) [14751.579705] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:17:53 (1788452273) === [14755.323072] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:17:56 (1788452276) === [14802.854351] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [14805.556412] Key type lgssc unregistered [14805.917340] LNet: 414871:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14805.926332] LNetError: 414871:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14805.942937] LNet: Removed LNI 192.168.204.156@tcp [14806.857273] Key type .llcrypt unregistered [14806.860630] Key type ._llcrypt unregistered