[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 518968263 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2464MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002000] x2apic enabled [ 0.002010] Switched APIC routing to physical x2apic. [ 0.003011] kvm-guest: setup PV IPIs [ 0.005792] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.006027] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.007019] pid_max: default: 32768 minimum: 301 [ 0.008147] LSM: Security Framework initializing [ 0.009049] Yama: becoming mindful. [ 0.010035] SELinux: Initializing. [ 0.011060] *** VALIDATE selinux *** [ 0.019508] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024164] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025128] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027025] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028154] *** VALIDATE tmpfs *** [ 0.030202] *** VALIDATE proc *** [ 0.031169] *** VALIDATE cgroup *** [ 0.032005] *** VALIDATE cgroup2 *** [ 0.033225] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034150] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036018] Spectre V2 : User space: Vulnerable [ 0.037004] Speculative Store Bypass: Vulnerable [ 0.040473] debug: unmapping init [mem 0xffffffffa2c59000-0xffffffffa2c60fff] [ 0.042176] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043751] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044028] ... version: 2 [ 0.045017] ... bit width: 48 [ 0.046013] ... generic registers: 4 [ 0.047014] ... value mask: 0000ffffffffffff [ 0.048015] ... max period: 00007fffffffffff [ 0.049017] ... fixed-purpose events: 3 [ 0.050012] ... event mask: 000000070000000f [ 0.051369] rcu: Hierarchical SRCU implementation. [ 0.053498] smp: Bringing up secondary CPUs ... [ 0.054538] x86: Booting SMP configuration: [ 0.055022] .... node #0, CPUs: #1 #2 #3 [ 0.063073] smp: Brought up 1 node, 4 CPUs [ 0.065011] smpboot: Max logical packages: 1 [ 0.066011] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.180000] node 0 deferred pages initialised in 112ms [ 0.183142] devtmpfs: initialized [ 0.184219] x86/mm: Memory block size: 128MB [ 0.188226] gcov: version magic: 0x41383552 [ 0.191354] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.192060] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.194171] pinctrl core: initialized pinctrl subsystem [ 0.196172] [ 0.196910] ************************************************************* [ 0.200013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.202010] ** ** [ 0.205023] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.208012] ** ** [ 0.211013] ** This means that this kernel is built to expose internal ** [ 0.214012] ** IOMMU data structures, which may compromise security on ** [ 0.217011] ** your system. ** [ 0.220012] ** ** [ 0.222017] ** If you see this message and you are not debugging the ** [ 0.224010] ** kernel, report this immediately to your vendor! ** [ 0.227013] ** ** [ 0.229011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.231010] ************************************************************* [ 0.234210] NET: Registered protocol family 16 [ 0.236752] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.239060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.242065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.246082] cpuidle: using governor menu [ 0.248900] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.251647] PCI: Using configuration type 1 for base access [ 0.253153] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.262053] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.264072] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.268127] cryptd: max_cpu_qlen set to 1000 [ 0.271252] ACPI: Added _OSI(Module Device) [ 0.273015] ACPI: Added _OSI(Processor Device) [ 0.275012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.277014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.284359] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.292461] ACPI: Interpreter enabled [ 0.295059] ACPI: PM: (supports S0 S3 S4 S5) [ 0.296011] ACPI: Using IOAPIC for interrupt routing [ 0.297136] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.300614] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.313146] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.314037] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.316019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.318078] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.322445] acpiphp: Slot [2] registered [ 0.323093] acpiphp: Slot [5] registered [ 0.325093] acpiphp: Slot [6] registered [ 0.326084] acpiphp: Slot [7] registered [ 0.327084] acpiphp: Slot [8] registered [ 0.328086] acpiphp: Slot [9] registered [ 0.329082] acpiphp: Slot [10] registered [ 0.331100] acpiphp: Slot [3] registered [ 0.332071] acpiphp: Slot [4] registered [ 0.333076] acpiphp: Slot [11] registered [ 0.334083] acpiphp: Slot [12] registered [ 0.335077] acpiphp: Slot [13] registered [ 0.336081] acpiphp: Slot [14] registered [ 0.338100] acpiphp: Slot [15] registered [ 0.339077] acpiphp: Slot [16] registered [ 0.340075] acpiphp: Slot [17] registered [ 0.341071] acpiphp: Slot [18] registered [ 0.343151] acpiphp: Slot [19] registered [ 0.344081] acpiphp: Slot [20] registered [ 0.345111] acpiphp: Slot [21] registered [ 0.346079] acpiphp: Slot [22] registered [ 0.347145] acpiphp: Slot [23] registered [ 0.349072] acpiphp: Slot [24] registered [ 0.350107] acpiphp: Slot [25] registered [ 0.351080] acpiphp: Slot [26] registered [ 0.352170] acpiphp: Slot [27] registered [ 0.354094] acpiphp: Slot [28] registered [ 0.355073] acpiphp: Slot [29] registered [ 0.356069] acpiphp: Slot [30] registered [ 0.357070] acpiphp: Slot [31] registered [ 0.358048] PCI host bridge to bus 0000:00 [ 0.359015] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.361037] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.363024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.365023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.368062] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.371021] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.372160] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.375715] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.377992] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.386014] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.391771] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.393022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.396019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.398017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.402024] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.404776] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.407035] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.409767] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.414013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.426807] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.430788] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.439100] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.448017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.457017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.473021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.484867] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.494018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.501013] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.515019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.523124] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.531017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.541018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.554015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.560752] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.569016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.576017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.590025] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.597430] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.604031] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.611026] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.630027] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.641246] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.652021] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.659016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.679017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.691392] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.693349] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.695660] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.697357] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.698123] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.703122] iommu: Default domain type: Passthrough [ 0.704349] SCSI subsystem initialized [ 0.705088] ACPI: bus type USB registered [ 0.706072] usbcore: registered new interface driver usbfs [ 0.707057] usbcore: registered new interface driver hub [ 0.709051] usbcore: registered new device driver usb [ 0.710119] pps_core: LinuxPPS API ver. 1 registered [ 0.711054] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.713043] PTP clock support registered [ 0.715041] EDAC MC: Ver: 3.0.0 [ 0.716281] PCI: Using ACPI for IRQ routing [ 0.717582] NetLabel: Initializing [ 0.718008] NetLabel: domain hash size = 128 [ 0.718958] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.719081] NetLabel: unlabeled traffic allowed by default [ 0.720351] vgaarb: loaded [ 0.721232] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.722012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.728197] clocksource: Switched to clocksource kvm-clock [ 0.828264] VFS: Disk quotas dquot_6.6.0 [ 0.829363] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.832490] *** VALIDATE ramfs *** [ 0.833778] *** VALIDATE hugetlbfs *** [ 0.835451] pnp: PnP ACPI init [ 0.838079] pnp: PnP ACPI: found 6 devices [ 0.851280] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.854787] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.857124] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.859326] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.861438] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.863514] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.866046] NET: Registered protocol family 2 [ 0.868136] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.871651] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.873956] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.878067] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.881230] TCP: Hash tables configured (established 65536 bind 65536) [ 0.883177] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.885190] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.887029] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.888907] NET: Registered protocol family 1 [ 0.890799] RPC: Registered named UNIX socket transport module. [ 0.892487] RPC: Registered udp transport module. [ 0.894383] RPC: Registered tcp transport module. [ 0.895932] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.897357] NET: Registered protocol family 44 [ 0.898539] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.899799] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.901346] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.903276] PCI: CLS 0 bytes, default 64 [ 0.904551] Unpacking initramfs... [ 2.303748] debug: unmapping init [mem 0xffff890cfcc54000-0xffff890cfffbffff] [ 2.307209] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.310391] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 2.312745] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.789826] Initialise system trusted keyrings [ 2.791587] Key type blacklist registered [ 2.793700] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.802300] zbud: loaded [ 2.805854] *** VALIDATE nfs *** [ 2.807197] *** VALIDATE nfs4 *** [ 2.808793] pstore: using deflate compression [ 2.812405] Platform Keyring initialized [ 2.907501] NET: Registered protocol family 38 [ 2.912519] Key type asymmetric registered [ 2.914295] Asymmetric key parser 'x509' registered [ 2.916409] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.918919] io scheduler mq-deadline registered [ 2.920221] io scheduler kyber registered [ 2.921699] io scheduler bfq registered [ 2.924144] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.927134] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.930035] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.932834] ACPI: Power Button [PWRF] [ 2.938162] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.945107] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.019715] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.031138] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.060613] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.086191] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.112049] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.116251] Non-volatile memory driver v1.3 [ 3.117507] Linux agpgart interface v0.103 [ 3.144342] virtio_blk virtio1: [vda] 145800 512-byte logical blocks (74.6 MB/71.2 MiB) [ 3.146372] vda: detected capacity change from 0 to 74649600 [ 3.169967] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.172816] vdb: detected capacity change from 0 to 1073741824 [ 3.191261] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.195142] vdc: detected capacity change from 0 to 2621440000 [ 3.213797] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.217293] vdd: detected capacity change from 0 to 2621440000 [ 3.232497] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.235242] vde: detected capacity change from 0 to 4294967296 [ 3.253122] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.255834] vdf: detected capacity change from 0 to 4294967296 [ 3.265117] libphy: Fixed MDIO Bus: probed [ 3.274854] usbcore: registered new interface driver usbserial_generic [ 3.276605] usbserial: USB Serial support registered for generic [ 3.278467] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.282693] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.284355] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.287164] mousedev: PS/2 mouse device common for all mice [ 3.290063] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.292577] rtc_cmos 00:05: RTC can wake from S4 [ 3.296232] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.299655] rtc_cmos 00:05: registered as rtc0 [ 3.302396] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.304365] intel_pstate: CPU model not supported [ 3.305357] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.314501] hid: raw HID events driver (C) Jiri Kosina [ 3.316367] usbcore: registered new interface driver usbhid [ 3.318049] usbhid: USB HID core driver [ 3.319267] drop_monitor: Initializing network drop monitor service [ 3.321088] Initializing XFRM netlink socket [ 3.322787] NET: Registered protocol family 10 [ 3.325133] Segment Routing with IPv6 [ 3.326220] NET: Registered protocol family 17 [ 3.328447] mpls_gso: MPLS GSO support [ 3.333783] RAS: Correctable Errors collector initialized. [ 3.335979] AVX version of gcm_enc/dec engaged. [ 3.337546] AES CTR mode by8 optimization enabled [ 3.410841] sched_clock: Marking stable (3410824468, 0)->(4271054681, -860230213) [ 3.413912] registered taskstats version 1 [ 3.416285] Loading compiled-in X.509 certificates [ 3.418451] zswap: loaded using pool lzo/zbud [ 3.445512] Key type big_key registered [ 3.456291] Key type encrypted registered [ 3.458272] ima: No TPM chip found, activating TPM-bypass! [ 3.460192] ima: Allocated hash algorithm: sha1 [ 3.461755] ima: No architecture policies found [ 3.463304] evm: Initialising EVM extended attributes: [ 3.464918] evm: security.selinux [ 3.466139] evm: security.ima [ 3.467350] evm: security.capability [ 3.468805] evm: HMAC attrs: 0x1 [ 3.471134] rtc_cmos 00:05: setting system clock to 2026-07-26 14:58:09 UTC (1785077889) [ 3.475380] debug: unmapping init [mem 0xffffffffa3c03000-0xffffffffa3dfffff] [ 3.477193] debug: unmapping init [mem 0xffffffffa2982000-0xffffffffa2c58fff] [ 3.488123] Write protecting the kernel read-only data: 28672k [ 3.491407] debug: unmapping init [mem 0xffffffffa1003000-0xffffffffa11fffff] [ 3.493778] debug: unmapping init [mem 0xffffffffa1914000-0xffffffffa19fffff] [ 3.533755] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.540783] systemd[1]: Detected virtualization kvm. [ 3.542431] systemd[1]: Detected architecture x86-64. [ 3.543953] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.568113] systemd[1]: No hostname configured. [ 3.569312] systemd[1]: Set hostname to . [ 3.571343] random: systemd: uninitialized urandom read (16 bytes read) [ 3.573492] systemd[1]: Initializing machine ID from random generator. [ 3.633933] random: ln: uninitialized urandom read (6 bytes read) [ 3.715182] random: systemd: uninitialized urandom read (16 bytes read) [ 3.718340] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.723923] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.726897] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Reached target Timers. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.848727] device-mapper: uevent: version 1.0.3 [ 4.851589] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.620980] virtio_net virtio0 ens2: renamed from eth0 [ 5.652557] random: fast init done [ 5.666168] scsi host0: ata_piix [ 5.695590] scsi host1: ata_piix [ 5.697152] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.699018] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.973030] random: crng init done [ 9.974431] random: 7 urandom warning(s) missed due to ratelimiting [ 10.109058] dracut-initqueue[595]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.770925] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.051749] printk: systemd: 23 output lines suppressed due to ratelimiting [ 12.288801] SELinux: Disabled at runtime. [ 12.345383] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.350991] systemd[1]: Detected virtualization kvm. [ 12.352141] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.859969] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.862884] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.867877] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.872551] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.876602] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.886407] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.894681] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Paths. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target Local Encrypted Volumes. Mounting Kernel Debug File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-getty.slice. [FAILED] Failed to [ 13.043634] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Huge Pages File System... Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 13.525724] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.822721] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.000574] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.052792] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.060875] EDAC sbridge: Ver: 1.1.2 [ 15.635425] Key type dns_resolver registered [ 15.936888] NFS: Registering the id_resolver key type [ 15.939255] Key type id_resolver registered [ 15.940980] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started dnf makecache --timer. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg120-server login: [ 53.458030] hrtimer: interrupt took 3035183 ns [ 75.307219] libcfs: loading out-of-tree module taints kernel. [ 75.407336] Key type ._llcrypt registered [ 75.412477] Key type .llcrypt registered [ 75.591393] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 94.033108] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 96.578407] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 96.626503] alg: No test for adler32 (adler32-zlib) [ 98.440233] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 99.723659] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 101.639184] Key type lgssc registered [ 104.682755] Lustre: Echo OBD driver; http://www.lustre.org/ [ 123.085302] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 170.938905] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 185.759102] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 185.813552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 187.064294] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 187.108602] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 187.272922] Lustre: lustre-MDT0000: new disk, initializing [ 187.376774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 187.405330] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 191.512821] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 205.534693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 205.660595] Lustre: 6533:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 205.695961] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 205.701903] Lustre: Skipped 1 previous similar message [ 205.822809] Lustre: lustre-MDT0001: new disk, initializing [ 205.894960] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 205.920232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 205.935260] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 210.691419] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 215.443548] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 224.633650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 224.874960] Lustre: lustre-OST0000: new disk, initializing [ 224.885472] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 224.893416] Lustre: 8469:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 224.969105] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 230.920449] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 234.559398] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 234.562555] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 234.676395] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 244.893324] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 245.020651] Lustre: lustre-OST0001: new disk, initializing [ 245.024150] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 245.027564] Lustre: 9544:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 245.125473] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 251.223567] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 255.014546] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 255.026875] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 255.117447] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 262.213918] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 266.195631] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 272.744643] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing check_logdir /tmp/testlogs/ [ 277.481837] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing yml_node [ 282.123847] Lustre: DEBUG MARKER: Client: 2.17.54.162 [ 284.611901] Lustre: DEBUG MARKER: MDS: 2.17.54.162 [ 287.038570] Lustre: DEBUG MARKER: OSS: 2.17.54.162 [ 288.558555] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Jul 26 11:02:53 EDT 2026 [ 307.388215] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 309.006383] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 326.626901] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 326.633282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 326.640605] Lustre: Skipped 2 previous similar messages [ 326.656139] Lustre: Skipped 3 previous similar messages [ 331.772093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 331.793455] Lustre: Skipped 3 previous similar messages [ 333.074774] Lustre: server umount lustre-MDT0000 complete [ 337.335377] LustreError: 6525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785078223 with bad export cookie 8030964842562592183 [ 337.336268] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 337.356627] LustreError: 6525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 341.986851] LustreError: 6539:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 341.991713] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 342.002184] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 342.021553] LustreError: 6539:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 342.037549] Lustre: Skipped 2 previous similar messages [ 343.008427] LustreError: 6540:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 343.040435] LustreError: 6540:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 343.961889] Lustre: server umount lustre-MDT0001 complete [ 348.855494] Lustre: server umount lustre-OST0000 complete [ 352.923711] Lustre: server umount lustre-OST0001 complete [ 360.787372] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 370.296955] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 406.489174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 406.857929] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 406.884404] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 407.036170] Lustre: lustre-MDT0000: new disk, initializing [ 407.193512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 407.216675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 411.884989] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 421.708111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 421.840395] Lustre: 15096:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 421.920601] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 421.927152] Lustre: Skipped 1 previous similar message [ 422.062205] Lustre: lustre-MDT0001: new disk, initializing [ 422.154643] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 422.197171] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 422.209702] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 426.011889] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 436.854487] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 445.011970] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 452.606984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 452.815189] Lustre: lustre-OST0000: new disk, initializing [ 452.819381] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 452.825499] Lustre: 17067:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 452.904662] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 454.437215] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 454.446554] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 454.578578] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 459.254690] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 469.060657] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 474.313936] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 474.534852] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 479.017480] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 479.215300] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 485.356227] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 485.369384] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 485.370241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 485.373522] LustreError: Skipped 1 previous similar message [ 485.382763] Lustre: Skipped 4 previous similar messages [ 487.258434] Lustre: server umount lustre-OST0000 complete [ 494.055964] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 494.068994] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 494.106410] Lustre: Skipped 1 previous similar message [ 494.118840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 496.967264] Lustre: server umount lustre-MDT0000 complete [ 500.714369] LustreError: 16555:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 500.745212] LustreError: 16555:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 500.815616] LustreError: 15903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785078386 with bad export cookie 8030964842562593716 [ 500.819832] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 500.824158] LustreError: 15903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 501.131703] Lustre: server umount lustre-MDT0001 complete [ 506.742786] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 11:06:31 (1785078391) [ 524.551260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 524.975591] LustreError: 19943:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 525.037741] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 529.312274] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 530.403567] LustreError: 19944:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 530.425981] LustreError: 19944:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 537.154809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 541.727659] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 548.527254] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 554.926892] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 557.357184] Lustre: Failing over lustre-MDT0000 [ 557.626750] Lustre: server umount lustre-MDT0000 complete [ 558.058706] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 558.063971] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 558.080979] Lustre: Skipped 1 previous similar message [ 558.091814] LustreError: 19943:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 558.114957] LustreError: 19943:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 576.003587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 576.078681] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 580.277128] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 581.601188] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 581.620106] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 581.639098] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 590.461986] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 596.966637] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 596.968290] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 596.987652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 596.996320] Lustre: Skipped 2 previous similar messages [ 597.015924] Lustre: Skipped 3 previous similar messages [ 602.085473] LustreError: 19940:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 602.093569] LustreError: 19940:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 602.692282] Lustre: server umount lustre-MDT0000 complete [ 605.542743] LustreError: 19924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785078491 with bad export cookie 8030964842562595130 [ 605.548183] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 605.550349] LustreError: 19924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 605.771749] Lustre: server umount lustre-MDT0001 complete [ 614.444693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 614.903640] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 614.908805] Lustre: Skipped 2 previous similar messages [ 620.941882] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 631.903745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 637.011922] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 645.591032] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 652.299311] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 654.203717] Lustre: Failing over lustre-MDT0000 [ 654.370288] Lustre: server umount lustre-MDT0000 complete [ 657.318992] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 657.333676] LustreError: 23274:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 657.343483] LustreError: 23274:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 672.516872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 672.594300] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 672.856824] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 677.877774] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 677.904781] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 677.917557] Lustre: Skipped 1 previous similar message [ 677.940504] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 678.064830] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 689.395136] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 697.312330] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 697.333654] Lustre: Skipped 1 previous similar message [ 697.337221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 699.987246] Lustre: server umount lustre-MDT0000 complete [ 703.390260] LustreError: 25942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785078589 with bad export cookie 8030964842562596068 [ 703.391064] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 703.397135] LustreError: 25942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 703.663657] Lustre: server umount lustre-MDT0001 complete [ 712.020815] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 11:09:56 (1785078596) [ 727.209829] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 736.263504] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 767.527201] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 778.619920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 778.925252] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 778.982506] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 779.083331] Lustre: lustre-MDT0000: new disk, initializing [ 779.178735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 779.185247] Lustre: Skipped 2 previous similar messages [ 779.210403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 784.247602] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 794.644290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 794.768309] Lustre: 29652:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 795.057756] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 799.988876] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 808.563376] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 813.582819] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 820.173485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 820.577440] Lustre: 31621:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 822.173490] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 822.208244] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 822.256524] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 822.269812] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 827.315864] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 836.541720] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 840.917416] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 858.088469] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 858.096677] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 858.111872] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 858.121929] Lustre: Skipped 2 previous similar messages [ 858.139592] Lustre: Skipped 2 previous similar messages [ 861.312766] Lustre: server umount lustre-OST0000 complete [ 866.785257] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 866.792025] LustreError: Skipped 1 previous similar message [ 871.268987] Lustre: server umount lustre-MDT0000 complete [ 873.442035] LustreError: 29665:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 873.484989] LustreError: 29665:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 874.598550] LustreError: 32825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785078760 with bad export cookie 8030964842562596474 [ 874.602520] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 874.609662] LustreError: 32825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 874.874113] Lustre: server umount lustre-MDT0001 complete [ 884.065947] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 886.601936] Key type lgssc unregistered [ 886.817688] LNet: 33576:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 886.822171] LNetError: 33576:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 886.833275] LNet: Removed LNI 192.168.201.120@tcp [ 887.621201] Key type .llcrypt unregistered [ 887.624765] Key type ._llcrypt unregistered [ 910.858212] Key type ._llcrypt registered [ 910.859887] Key type .llcrypt registered [ 910.974571] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 912.061701] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 912.143871] alg: No test for adler32 (adler32-zlib) [ 913.219493] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 913.527639] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 915.223331] Key type lgssc registered [ 916.263055] Lustre: Echo OBD driver; http://www.lustre.org/ [ 922.729899] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 922.750063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 924.295731] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 928.784558] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 934.882544] LustreError: 34993:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 934.941427] LustreError: 34993:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 936.803117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 937.054698] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 941.821884] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 950.306224] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 957.064873] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 964.640770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 964.901911] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 969.031871] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 969.320962] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 976.105173] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 982.314057] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 992.193917] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 11:14:37 (1785078877) [ 993.719460] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 994.977222] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1000.613977] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 11:14:45 (1785078885) [ 1008.918628] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 11:14:53 (1785078893) [ 1009.876748] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1017.730355] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1017.735642] Lustre: Skipped 2 previous similar messages [ 1019.551887] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 11:15:04 (1785078904) [ 1022.740737] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1022.744110] Lustre: Skipped 3 previous similar messages [ 1038.921394] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1038.925763] Lustre: Skipped 6 previous similar messages [ 1041.471872] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 11:15:26 (1785078926) [ 1044.146669] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1047.167585] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1054.377870] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 11:15:39 (1785078939) [ 1055.846374] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1055.850771] Lustre: Skipped 3 previous similar messages [ 1065.935282] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 11:15:50 (1785078950) [ 1079.776682] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1079.789637] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1079.800855] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1081.826216] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1081.843749] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1081.870077] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1084.148377] Lustre: server umount lustre-OST0000 complete [ 1087.968900] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1087.985418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1087.992829] Lustre: Skipped 3 previous similar messages [ 1093.089548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1093.099891] Lustre: Skipped 1 previous similar message [ 1093.684309] Lustre: server umount lustre-MDT0000 complete [ 1096.881876] LustreError: 38551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785078982 with bad export cookie 12158011192064906071 [ 1096.882832] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1096.911368] LustreError: 38551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1097.311358] Lustre: server umount lustre-MDT0001 complete [ 1106.424950] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 1108.291385] Key type lgssc unregistered [ 1108.498537] LNet: 42842:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1108.506408] LNetError: 42842:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1109.545450] LNet: Removed LNI 192.168.201.120@tcp [ 1110.390154] Key type .llcrypt unregistered [ 1110.391482] Key type ._llcrypt unregistered [ 1126.545285] Key type ._llcrypt registered [ 1126.546920] Key type .llcrypt registered [ 1126.627153] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 1140.193177] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 1141.063755] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1141.142672] alg: No test for adler32 (adler32-zlib) [ 1142.152635] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 1142.364116] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 1144.031324] Key type lgssc registered [ 1145.252199] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1178.034844] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1178.068973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1179.316286] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1179.373990] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1179.481455] Lustre: lustre-MDT0000: new disk, initializing [ 1179.603519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1179.630043] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1184.532675] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1196.268638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1196.358492] Lustre: 45984:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1196.428749] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1196.433115] Lustre: Skipped 1 previous similar message [ 1196.495252] Lustre: lustre-MDT0001: new disk, initializing [ 1196.600602] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1196.633682] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1196.643770] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1201.555304] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1210.745917] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1216.941288] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1223.203749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1223.391770] Lustre: lustre-OST0000: new disk, initializing [ 1223.393517] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1223.395976] Lustre: 47956:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1223.442951] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1229.348273] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1233.541979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1233.551308] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1233.681804] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1238.698312] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1243.007749] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1243.226666] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1249.172996] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1249.388537] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1254.373356] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1254.380069] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1254.389357] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1254.394890] Lustre: Skipped 1 previous similar message [ 1257.691215] Lustre: server umount lustre-OST0000 complete [ 1263.072974] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1263.081083] LustreError: Skipped 1 previous similar message [ 1263.084356] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1263.102160] Lustre: Skipped 1 previous similar message [ 1263.107590] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1264.607937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1264.614049] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1267.901663] Lustre: server umount lustre-MDT0000 complete [ 1272.155201] LustreError: 45978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785079158 with bad export cookie 15174483319827241868 [ 1272.166125] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1272.166371] LustreError: 45978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1272.676431] Lustre: server umount lustre-MDT0001 complete [ 1282.167256] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 11:19:27 (1785079167) [ 1288.558053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1289.018359] LustreError: 50415:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1289.038703] LustreError: 50415:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1289.114992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1292.953349] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1294.312227] LustreError: 50416:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1299.425489] LustreError: 50415:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1300.866091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1301.217447] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1305.574978] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1312.799807] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1318.817560] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1326.429536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1326.748748] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1334.031463] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1341.642801] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1350.420807] LustreError: 52945:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1351.210364] Lustre: Modifying parameter general.timeout=1 in log params [ 1352.267043] Lustre: Modifying parameter general.timeout=2 in log params [ 1355.113906] Lustre: Modifying parameter general.timeout=5 in log params [ 1355.117163] Lustre: Skipped 2 previous similar messages [ 1359.306407] Lustre: Modifying parameter general.timeout=10 in log params [ 1359.311411] Lustre: Skipped 4 previous similar messages [ 1367.379575] Lustre: Modifying parameter general.timeout=20 in log params [ 1367.385612] Lustre: Skipped 9 previous similar messages [ 1383.570090] Lustre: Modifying parameter general.timeout=39 in log params [ 1383.575123] Lustre: Skipped 18 previous similar messages [ 1415.717860] Lustre: Modifying parameter general.timeout=80 in log params [ 1415.726200] Lustre: Skipped 40 previous similar messages [ 1465.126786] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 11:22:29 (1785079349) [ 1467.862991] Lustre: 59874:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1469.551914] Lustre: 59973:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1471.561258] Lustre: 60075:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1477.103598] Lustre: 60383:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1477.111991] Lustre: 60383:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1480.733468] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 in log params [ 1480.747913] Lustre: Skipped 58 previous similar messages [ 1505.701995] Lustre: 61736:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1505.733972] LustreError: 61736:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1511.070335] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 11:23:15 (1785079395) [ 1526.757413] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1526.762328] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1526.795361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1526.798992] Lustre: Skipped 1 previous similar message [ 1530.679659] Lustre: server umount lustre-MDT0000 complete [ 1533.411737] LustreError: 51143:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1533.428569] LustreError: 51143:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1534.449603] LustreError: 50395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785079420 with bad export cookie 15174483319827242750 [ 1534.457720] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1534.460581] LustreError: 50395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1534.945264] Lustre: server umount lustre-MDT0001 complete [ 1549.252393] Lustre: server umount lustre-OST0000 complete [ 1574.285383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1574.671781] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1575.193858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1579.287263] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1587.182081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1587.260891] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1587.311345] Lustre: 64270:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1591.734584] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1598.785810] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1603.930355] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1609.875408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1610.082438] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1610.360975] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1610.363511] Lustre: Skipped 1 previous similar message [ 1616.364680] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1623.440817] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1624.731187] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1624.736770] Lustre: Skipped 14 previous similar messages [ 1624.767391] Lustre: 66836:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1634.797437] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1634.811063] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1634.822902] Lustre: Skipped 2 previous similar messages [ 1634.832435] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1634.834905] Lustre: Skipped 3 previous similar messages [ 1636.615403] Lustre: server umount lustre-OST0000 complete [ 1644.010677] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1644.022304] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1644.036281] Lustre: Skipped 1 previous similar message [ 1644.043356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1644.047884] Lustre: Skipped 2 previous similar messages [ 1646.823483] Lustre: server umount lustre-MDT0000 complete [ 1650.231412] LustreError: 64261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785079536 with bad export cookie 15174483319827245676 [ 1650.255986] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1650.599229] Lustre: server umount lustre-MDT0001 complete [ 1660.839583] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 1663.331295] Key type lgssc unregistered [ 1663.607797] LNet: 68045:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1663.614770] LNetError: 68045:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1664.677779] LNet: Removed LNI 192.168.201.120@tcp [ 1665.565103] Key type .llcrypt unregistered [ 1665.572695] Key type ._llcrypt unregistered [ 1687.822314] Key type ._llcrypt registered [ 1687.823882] Key type .llcrypt registered [ 1687.898398] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 1688.524051] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1688.624673] alg: No test for adler32 (adler32-zlib) [ 1689.631071] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 1689.900831] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 1691.551159] Key type lgssc registered [ 1693.751441] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1704.715941] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1704.744997] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1706.709923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1711.924944] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1716.198200] LustreError: 69461:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1716.218925] LustreError: 69461:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1720.192030] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1720.729866] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1725.398742] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1732.092991] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1738.397739] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1746.161345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1746.915979] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1753.843863] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1761.788183] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1773.788231] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1780.710527] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1780.715682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1780.726356] Lustre: Skipped 2 previous similar messages [ 1780.744492] Lustre: Skipped 2 previous similar messages [ 1785.835959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1785.857037] Lustre: Skipped 2 previous similar messages [ 1786.837026] Lustre: server umount lustre-MDT0000 complete [ 1790.948253] LustreError: 69460:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1790.977353] LustreError: 69460:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1791.823499] LustreError: 69441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785079677 with bad export cookie 3171999708204178843 [ 1791.832902] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1791.836030] LustreError: 69441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1792.561772] Lustre: server umount lustre-MDT0001 complete [ 1807.755694] Lustre: server umount lustre-OST0000 complete [ 1831.049853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1831.343411] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1831.359383] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1831.612406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1835.520683] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1842.968277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1843.142857] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1843.168847] Lustre: 74332:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1843.557166] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1847.919490] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1855.548963] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1861.151682] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1868.024247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1868.286829] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1868.735376] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1875.246435] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1882.802971] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1884.191063] Lustre: 76908:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1886.041369] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1891.313714] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1891.325089] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1891.349403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1896.421705] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1896.430741] Lustre: Skipped 2 previous similar messages [ 1897.523820] Lustre: server umount lustre-OST0000 complete [ 1901.537622] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1901.541237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1901.552803] Lustre: Skipped 2 previous similar messages [ 1901.579690] Lustre: Skipped 1 previous similar message [ 1907.523242] Lustre: server umount lustre-MDT0000 complete [ 1911.009641] LustreError: 76199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785079797 with bad export cookie 3171999708204179683 [ 1911.010716] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1911.016718] LustreError: 76199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 1911.312239] Lustre: server umount lustre-MDT0001 complete [ 1922.830742] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 1925.779249] Key type lgssc unregistered [ 1926.164222] LNet: 78120:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1926.176645] LNetError: 78120:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1926.200494] LNet: Removed LNI 192.168.201.120@tcp [ 1926.980598] Key type .llcrypt unregistered [ 1926.982402] Key type ._llcrypt unregistered [ 1949.666911] Key type ._llcrypt registered [ 1949.668933] Key type .llcrypt registered [ 1949.782895] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 1951.090811] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1951.104044] alg: No test for adler32 (adler32-zlib) [ 1952.114348] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 1952.377689] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 1954.031388] Key type lgssc registered [ 1955.282878] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1963.160354] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1963.179930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1964.831379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1969.785280] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1974.242873] LustreError: 79534:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1974.274569] LustreError: 79534:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1978.883996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1979.595506] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1984.678374] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1991.633115] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1997.267425] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2003.620706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2004.103318] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2010.650493] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2017.782995] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2024.309097] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2025.097387] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2026.597184] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2026.603427] Lustre: Skipped 1 previous similar message [ 2029.017356] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2029.024865] Lustre: Skipped 2 previous similar messages [ 2033.056514] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2033.063899] Lustre: Skipped 4 previous similar messages [ 2041.179951] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2041.183384] Lustre: Skipped 8 previous similar messages [ 2057.937535] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2057.942211] Lustre: Skipped 17 previous similar messages [ 2090.404947] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2090.412304] Lustre: Skipped 36 previous similar messages [ 2119.203713] Lustre: 87634:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2121.981919] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 11:33:26 (1785080006) [ 2123.666815] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2125.708850] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 11:33:30 (1785080010) [ 2132.449906] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2132.452689] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2132.457522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2132.493466] Lustre: Skipped 1 previous similar message [ 2136.441594] Lustre: server umount lustre-OST0000 complete [ 2142.691762] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2142.698991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2142.712783] Lustre: Skipped 1 previous similar message [ 2142.715630] Lustre: Skipped 1 previous similar message [ 2146.357621] Lustre: server umount lustre-MDT0000 complete [ 2147.808896] LustreError: 79535:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2147.827033] LustreError: 79535:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2150.932344] LustreError: 79514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785080036 with bad export cookie 16090831640175336542 [ 2150.935701] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2150.943526] LustreError: 79514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2151.298930] Lustre: server umount lustre-MDT0001 complete [ 2164.256382] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 2166.966582] Key type lgssc unregistered [ 2167.180762] LNet: 88882:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2167.195270] LNetError: 88882:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2168.232202] LNet: Removed LNI 192.168.201.120@tcp [ 2169.256843] Key type .llcrypt unregistered [ 2169.259119] Key type ._llcrypt unregistered [ 2174.925744] Key type ._llcrypt registered [ 2174.930987] Key type .llcrypt registered [ 2175.046974] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2180.320682] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules [ 2181.904613] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2181.920682] alg: No test for adler32 (adler32-zlib) [ 2182.924494] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 2183.145505] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 2184.847206] Key type lgssc registered [ 2186.755517] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2188.270663] LustreError: 90093:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2205.935515] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 2209.808182] LustreError: 90093:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2217.078643] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2217.094146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2218.571851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2222.889224] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2228.706566] LustreError: 91105:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2228.717490] LustreError: 91105:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2230.784881] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 11:35:15 (1785080115) [ 2233.825222] LustreError: 91104:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2235.774256] LustreError: 91136:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2235.792766] LustreError: 91136:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2235.800777] LustreError: 91136:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 0, failed: rc = -5 [ 2236.310879] Lustre: server umount lustre-MDT0000 complete [ 2253.577292] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 2256.926838] Key type lgssc unregistered [ 2257.222919] LNet: 92630:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2257.228762] LNetError: 92630:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2258.277370] LNet: Removed LNI 192.168.201.120@tcp [ 2259.443963] Key type .llcrypt unregistered [ 2259.447171] Key type ._llcrypt unregistered [ 2282.113358] Key type ._llcrypt registered [ 2282.122584] Key type .llcrypt registered [ 2282.236479] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 2283.650117] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2283.692193] alg: No test for adler32 (adler32-zlib) [ 2284.830656] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 2285.204393] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 2286.927820] Key type lgssc registered [ 2288.345646] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2297.671657] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2297.697806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2299.610837] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2304.533074] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2309.089574] LustreError: 93857:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2309.119765] LustreError: 93857:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2313.094365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2313.848325] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2318.408593] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2326.431538] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2333.373973] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2340.814834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2341.434659] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2348.157207] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2356.371624] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2405.757113] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 11:38:10 (1785080290) [ 2407.398256] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2409.644215] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 11:38:14 (1785080294) [ 2415.075932] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2415.080182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2415.095800] Lustre: Skipped 2 previous similar messages [ 2415.115562] Lustre: Skipped 2 previous similar messages [ 2420.196316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2420.212640] Lustre: Skipped 2 previous similar messages [ 2420.367254] Lustre: server umount lustre-MDT0000 complete [ 2424.121546] LustreError: 93838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785080310 with bad export cookie 2873384279128774707 [ 2424.131232] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2424.139418] LustreError: 93838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2424.568563] Lustre: server umount lustre-MDT0001 complete [ 2438.510485] Lustre: server umount lustre-OST0000 complete [ 2449.925952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2450.527425] LustreError: 98131:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2450.829253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2456.034234] LustreError: 98132:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2456.085900] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2460.133706] LustreError: 98131:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2463.986356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2464.836673] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2469.901791] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2477.780727] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2484.501199] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2499.917180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2500.132260] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2500.140695] LustreError: 98120:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2500.172538] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2500.184290] LustreError: 100214:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2500.199410] LustreError: 100214:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2500.221057] LustreError: 100214:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2500.335730] Lustre: server umount lustre-OST0000 complete [ 2500.338450] LustreError: 100214:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2510.255038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2510.560113] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2510.566893] LustreError: 98121:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2510.628431] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2510.640808] LustreError: 100677:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2510.653866] LustreError: 100677:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2510.659688] LustreError: 100677:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2510.738433] Lustre: server umount lustre-OST0000 complete [ 2510.741819] LustreError: 100677:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2521.852911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2522.176783] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2522.179865] Lustre: Found index 0 for lustre-OST0000, updating log [ 2522.190112] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2522.336995] Lustre: lustre-OST0000: new disk, initializing [ 2522.340230] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2522.344846] Lustre: 101216:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2522.682538] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2530.581298] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2534.751405] Lustre: Failing over lustre-OST0000 [ 2534.931517] Lustre: server umount lustre-OST0000 complete [ 2538.906481] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2538.935309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2538.938610] Lustre: Skipped 1 previous similar message [ 2544.096290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2544.102116] Lustre: Skipped 1 previous similar message [ 2549.218257] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2553.311261] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2553.497342] Lustre: server umount lustre-MDT0000 complete [ 2554.339762] LustreError: 98132:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2554.350518] LustreError: 98132:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2557.473184] LustreError: 98111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785080443 with bad export cookie 2873384279128777451 [ 2557.481470] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2557.487608] LustreError: 98111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2557.752235] Lustre: server umount lustre-MDT0001 complete [ 2567.363851] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 11:40:51 (1785080451) [ 2585.553578] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 2597.905181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2598.405294] LustreError: 103563:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2598.496890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2604.569996] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2614.570665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2620.976820] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2624.088926] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2631.787646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2632.876769] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2632.881866] Lustre: Skipped 1 previous similar message [ 2640.007837] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2648.969199] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2649.073864] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2649.152758] Lustre: lustre-OST0001: new disk, initializing [ 2649.160324] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2649.165386] Lustre: 105880:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2655.883493] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2655.892624] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2656.050829] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2656.556545] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2665.617500] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2673.312826] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2680.729161] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.20@tcp (stopping) [ 2680.732603] Lustre: Skipped 1 previous similar message [ 2681.832494] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2681.857851] Lustre: Skipped 2 previous similar messages [ 2684.434200] Lustre: server umount lustre-MDT0001 complete [ 2685.854129] LustreError: 103560:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2685.863549] LustreError: 103560:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2695.296655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2695.385658] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2695.391269] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2695.473181] Lustre: 103553:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2697.271126] LustreError: 107451:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2697.287673] LustreError: 107451:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2697.300903] LustreError: 103555:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2697.482694] LustreError: 107463:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.120@tcp: cfg command failed: rc = -17 [ 2697.492907] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2697.492907] [ 2701.092546] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2701.814630] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 2711.380877] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 11:43:16 (1785080596) [ 2730.899958] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 2753.073499] Lustre: 109500:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2778.702281] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2782.880106] Lustre: 110635:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2869.734123] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2869.758493] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2869.773683] Lustre: Skipped 1 previous similar message [ 2869.782641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2869.789762] Lustre: Skipped 3 previous similar messages [ 2873.719528] Lustre: server umount lustre-MDT0000 complete [ 2875.881672] LustreError: 103559:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2875.903261] LustreError: 103559:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 2877.687794] LustreError: 103545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785080763 with bad export cookie 2873384279128778438 [ 2877.694544] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2877.698399] LustreError: 103545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2877.969703] Lustre: server umount lustre-MDT0001 complete [ 2892.794487] Lustre: server umount lustre-OST0000 complete [ 2897.375132] Lustre: 93167:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785080767/real 1785080767] req@ffff890d442d6680 x1871792225747840/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785080783 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2897.450166] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2897.469809] Lustre: Skipped 3 previous similar messages [ 2898.009718] Lustre: server umount lustre-OST0001 complete [ 2906.189936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2920.191696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2930.424196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2941.439614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2963.940152] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 2976.802504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2976.841095] Lustre: lustre-MDT0000: reset Object Index mappings [ 2977.443770] LustreError: 114594:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2977.645146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2977.650730] Lustre: Skipped 2 previous similar messages [ 2983.113395] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2993.109448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2993.125113] Lustre: lustre-MDT0001: reset Object Index mappings [ 2999.008038] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 3003.435958] Lustre: 115746:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3010.487330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3018.186225] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:195 to 0x280000400:225) [ 3019.797110] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 3022.269838] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:354 to 0x280000401:386) [ 3032.132291] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3038.240883] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3038.252149] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3040.659175] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 3049.427070] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3053.665062] Lustre: 117623:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3105.654660] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 11:49:50 (1785080990) [ 3131.872317] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3131.880905] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3131.899956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3131.907846] Lustre: Skipped 4 previous similar messages [ 3137.170914] Lustre: server umount lustre-MDT0000 complete [ 3140.583535] LustreError: 114594:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3140.619253] LustreError: 114594:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 3141.077850] LustreError: 117625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785081027 with bad export cookie 2873384279128825352 [ 3141.114577] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3147.629875] Lustre: server umount lustre-MDT0001 complete [ 3166.175668] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3166.378429] Lustre: server umount lustre-OST0000 complete [ 3185.119491] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3185.365129] Lustre: server umount lustre-OST0001 complete [ 3193.316226] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 3205.695653] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 3247.333295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3247.597838] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3247.630657] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3247.755804] Lustre: lustre-MDT0000: new disk, initializing [ 3247.875913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3247.883720] Lustre: Skipped 3 previous similar messages [ 3247.900860] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3253.075214] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 3261.152804] Lustre: server umount lustre-MDT0000 complete [ 3271.614992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3271.863382] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3271.885914] Lustre: 121896:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3271.897372] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3276.520703] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 3281.692092] Lustre: server umount lustre-MDT0000 complete [ 3290.883333] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 11:52:55 (1785081175) [ 3293.063718] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3295.106738] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 11:52:59 (1785081179) [ 3314.776930] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 3326.849048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3332.594489] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 3342.788760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3342.929535] Lustre: 123950:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3342.977085] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3342.983718] Lustre: Skipped 1 previous similar message [ 3343.057736] Lustre: lustre-MDT0001: new disk, initializing [ 3343.204351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3343.225434] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3348.174787] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 3353.747334] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3361.015369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3361.331265] Lustre: lustre-OST0000: new disk, initializing [ 3361.337789] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3361.346688] Lustre: 125532:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3363.253168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3363.271083] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3363.423698] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3369.255689] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 3380.180112] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3380.424149] Lustre: lustre-OST0001: new disk, initializing [ 3380.435859] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3380.442213] Lustre: 126400:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3382.343184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3382.356930] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3382.458278] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3388.279320] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 3398.912515] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3402.127966] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3758.392262] Lustre: 132808:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3758.409733] Lustre: 132808:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3758.420569] Lustre: 132808:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3758.426662] Lustre: 132808:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3758.435793] Lustre: 132808:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3758.439937] Lustre: 132808:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4030.760670] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 12:05:14 (1785081914) [ 4046.317461] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4046.334564] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4046.335969] Lustre: Skipped 8 previous similar messages [ 4046.348810] Lustre: Skipped 10 previous similar messages [ 4052.257772] Lustre: server umount lustre-MDT0000 complete [ 4056.550275] LustreError: 127210:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4056.576346] LustreError: 127210:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 4056.714055] LustreError: 123940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785081942 with bad export cookie 2873384279128892769 [ 4056.722787] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4056.728447] LustreError: 123940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4061.671064] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4061.673189] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4061.679638] Lustre: Skipped 1 previous similar message [ 4061.711306] Lustre: Skipped 6 previous similar messages [ 4071.394080] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4071.878610] Lustre: server umount lustre-MDT0001 complete [ 4090.335111] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4090.561372] Lustre: server umount lustre-OST0000 complete [ 4109.279133] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4109.513458] Lustre: server umount lustre-OST0001 complete [ 4117.976944] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 4131.029178] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 4177.440567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4177.747898] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4177.776840] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4177.845584] Lustre: lustre-MDT0000: new disk, initializing [ 4177.968689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4177.977699] Lustre: Skipped 5 previous similar messages [ 4177.994367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4183.263273] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4195.389811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4195.593886] Lustre: 135869:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4195.677937] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4195.682170] Lustre: Skipped 1 previous similar message [ 4195.793805] Lustre: lustre-MDT0001: new disk, initializing [ 4195.951892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4195.958718] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4200.377899] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4209.941508] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4216.142072] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4223.218173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4223.486216] Lustre: lustre-OST0000: new disk, initializing [ 4223.490331] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4223.498499] Lustre: 137839:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4223.593805] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4223.603249] Lustre: Skipped 1 previous similar message [ 4225.521263] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4225.536542] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4225.660611] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4230.583114] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4240.657199] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4251.233965] Lustre: lustre-MDD0000: changelog on [ 4253.932224] Lustre: lustre-MDD0001: changelog on [ 5977.706543] Lustre: lustre-MDD0001: changelog off [ 5979.728473] Lustre: 135878:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 5980.817655] Lustre: lustre-MDD0000: changelog off [ 5986.788334] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5986.793696] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5986.796445] LustreError: Skipped 1 previous similar message [ 5986.798479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5986.798483] Lustre: Skipped 2 previous similar messages [ 5986.823021] Lustre: Skipped 1 previous similar message [ 5991.913685] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5991.925671] Lustre: Skipped 2 previous similar messages [ 5992.631410] Lustre: server umount lustre-OST0000 complete [ 5998.048701] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5998.061429] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5998.087635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6002.148153] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6003.615811] Lustre: server umount lustre-MDT0000 complete [ 6007.267195] LustreError: 135882:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6007.288780] LustreError: 135882:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6007.676777] LustreError: 135862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785083893 with bad export cookie 2873384279129709172 [ 6007.689556] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6007.690543] LustreError: 135862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6008.090916] Lustre: server umount lustre-MDT0001 complete [ 6022.109341] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 6025.228794] Key type lgssc unregistered [ 6025.581992] LNet: 146946:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6025.588846] LNetError: 146946:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6025.601730] LNet: Removed LNI 192.168.201.120@tcp [ 6026.826180] Key type .llcrypt unregistered [ 6026.833356] Key type ._llcrypt unregistered [ 6049.910874] Key type ._llcrypt registered [ 6049.913576] Key type .llcrypt registered [ 6050.057631] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 6068.388656] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 6070.080899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6070.106033] alg: No test for adler32 (adler32-zlib) [ 6071.304960] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 6071.621834] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 6073.391206] Key type lgssc registered [ 6074.819866] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6111.511685] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6111.541935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6112.846857] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6112.869917] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6112.933953] Lustre: lustre-MDT0000: new disk, initializing [ 6113.010117] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6113.027861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6118.355768] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6129.037965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6129.181711] Lustre: 150149:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6129.228134] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6129.232480] Lustre: Skipped 1 previous similar message [ 6129.285778] Lustre: lustre-MDT0001: new disk, initializing [ 6129.383056] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6129.427671] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6129.439847] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6134.085962] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6143.375617] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6149.417461] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6156.155101] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6156.476381] Lustre: lustre-OST0000: new disk, initializing [ 6156.481993] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6156.500708] Lustre: 152117:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6156.652990] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6163.271346] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6164.017426] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6164.028825] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6164.175219] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6173.428776] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6180.026218] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_module obdecho/obdecho [ 6182.030576] LustreError: 153119:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6182.036965] LustreError: 153119:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6183.796430] LustreError: 153215:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6196.192948] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6196.202642] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6196.230317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6200.290352] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6200.292306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6200.311960] Lustre: Skipped 1 previous similar message [ 6200.338168] Lustre: Skipped 2 previous similar messages [ 6204.059978] Lustre: server umount lustre-MDT0000 complete [ 6207.903127] LustreError: 150141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785084093 with bad export cookie 7663867694965571834 [ 6207.909755] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6207.912542] LustreError: 150141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6208.409632] Lustre: server umount lustre-MDT0001 complete [ 6222.588776] Lustre: server umount lustre-OST0000 complete [ 6245.567640] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 6258.645120] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 6295.956248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6296.246410] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6296.270983] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6296.346408] Lustre: lustre-MDT0000: new disk, initializing [ 6296.424278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6296.439979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6301.066846] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6311.165494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6311.247574] Lustre: 156933:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6311.273573] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6311.275365] Lustre: Skipped 1 previous similar message [ 6311.354351] Lustre: lustre-MDT0001: new disk, initializing [ 6311.418412] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6311.449557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6311.474251] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6316.312159] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6327.119578] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6334.938450] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6343.884922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6344.294761] Lustre: lustre-OST0000: new disk, initializing [ 6344.302288] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6344.314757] Lustre: 158903:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6344.466240] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6351.952597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6351.964146] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6352.137763] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6352.397184] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6362.866923] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6369.064751] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6369.303361] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6374.958072] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6375.261957] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6377.952929] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6377.962805] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6377.973320] LustreError: Skipped 1 previous similar message [ 6377.974042] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6378.020763] Lustre: Skipped 1 previous similar message [ 6381.104393] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6381.114935] Lustre: Skipped 2 previous similar messages [ 6384.204820] Lustre: server umount lustre-OST0000 complete [ 6388.208351] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6388.225270] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6388.241832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6394.417561] Lustre: server umount lustre-MDT0000 complete [ 6396.384913] LustreError: 156938:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6396.409960] LustreError: 156938:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6397.769047] LustreError: 158902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785084283 with bad export cookie 7663867694965572793 [ 6397.771177] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6397.784871] LustreError: 158902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6398.106040] Lustre: server umount lustre-MDT0001 complete [ 6401.624091] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 12:44:46 (1785084286) [ 6403.328161] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6404.872454] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 12:44:49 (1785084289) [ 6412.383256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6412.987218] LustreError: 161316:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6413.121591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6418.403495] LustreError: 161317:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6418.434535] LustreError: 161317:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6418.915825] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6423.538061] LustreError: 161316:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6428.529731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6428.640904] LustreError: 161317:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6433.888752] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6441.723332] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6447.938344] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6454.924070] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6455.269323] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6455.277662] Lustre: Skipped 1 previous similar message [ 6462.100727] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6469.660892] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6480.309391] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.20@tcp (stopping) [ 6480.311652] Lustre: Skipped 2 previous similar messages [ 6481.516426] Lustre: server umount lustre-MDT0000 complete [ 6483.941839] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6483.951671] Lustre: Skipped 2 previous similar messages [ 6483.952725] LustreError: 161317:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6483.972058] LustreError: 161317:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6485.174723] LustreError: 161296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785084371 with bad export cookie 7663867694965573675 [ 6485.195081] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6491.870334] Lustre: server umount lustre-MDT0001 complete [ 6498.952968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6513.619106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6513.944932] LustreError: 165159:0:(llog.c:1655:llog_backup()) MGC192.168.201.120@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6513.950389] Lustre: 165159:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.120@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6515.154678] LustreError: 165164:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6515.183525] LustreError: 165164:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 6550.496577] LustreError: 165164:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6550.515842] LustreError: 165164:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 28 previous similar messages [ 6574.559299] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6574.567455] Lustre: 165159:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6615.541768] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6615.542029] LustreError: 165165:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6615.597229] LustreError: 165165:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 44 previous similar messages [ 6620.652714] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6621.097141] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6631.216196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6631.980137] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x6a5b87104c03e161 [ 6631.995995] Lustre: MGC192.168.201.120@tcp: Connection restored to 0@lo (at 0@lo) [ 6637.567383] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6637.599358] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6645.945873] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6648.076980] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6655.894215] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6657.912731] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6673.379301] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6673.385348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6673.402482] Lustre: Skipped 4 previous similar messages [ 6673.422685] Lustre: Skipped 6 previous similar messages [ 6675.541283] Lustre: server umount lustre-MDT0000 complete [ 6679.797099] LustreError: 161296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785084565 with bad export cookie 7663867694965580129 [ 6679.800991] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6679.807633] LustreError: 161296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 6680.444289] Lustre: server umount lustre-MDT0001 complete [ 6695.073695] Lustre: server umount lustre-OST0000 complete [ 6720.049738] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 6733.625260] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 6776.271237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6776.614681] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6776.666166] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6776.866435] Lustre: lustre-MDT0000: new disk, initializing [ 6776.988208] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6777.005509] Lustre: Skipped 1 previous similar message [ 6777.054317] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6782.665350] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6794.775436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6794.935066] Lustre: 170268:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6795.350229] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6800.454253] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6811.256099] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6817.866963] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6825.429724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6825.875319] Lustre: lustre-OST0000: new disk, initializing [ 6825.878179] Lustre: Skipped 1 previous similar message [ 6825.881838] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6825.883824] Lustre: Skipped 2 previous similar messages [ 6825.889115] Lustre: 172233:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6833.394481] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6834.804913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6834.821670] Lustre: Skipped 1 previous similar message [ 6834.835924] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6834.990826] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6843.284884] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6848.957684] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6849.218235] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6854.385482] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6854.709897] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6860.774661] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6860.778970] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6860.797192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6871.012321] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6871.502611] Lustre: server umount lustre-OST0000 complete [ 6882.726128] Lustre: server umount lustre-MDT0000 complete [ 6886.370805] LustreError: 170278:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6886.393494] LustreError: 170278:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 19 previous similar messages [ 6887.648828] LustreError: 172242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785084773 with bad export cookie 7663867694965596971 [ 6887.662917] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6888.120245] Lustre: server umount lustre-MDT0001 complete [ 6896.921425] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 12:53:01 (1785084781) [ 6904.879093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6909.702616] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6920.283591] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6926.890945] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6936.081772] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6942.704823] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6950.262560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6958.670773] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6967.129650] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6977.909297] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 12:54:22 (1785084862) [ 6981.610142] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6981.621270] LustreError: Skipped 1 previous similar message [ 6981.629330] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6981.638359] Lustre: Skipped 3 previous similar messages [ 6981.653788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6981.668286] Lustre: Skipped 9 previous similar messages [ 6987.809553] Lustre: server umount lustre-OST0000 complete [ 6992.353115] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6992.360805] LustreError: Skipped 1 previous similar message [ 6998.329113] Lustre: server umount lustre-MDT0000 complete [ 7002.831898] LustreError: 174628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785084888 with bad export cookie 7663867694965597853 [ 7002.846883] LustreError: 174628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7002.847029] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7003.249381] Lustre: server umount lustre-MDT0001 complete [ 7015.709759] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 7018.940671] Key type lgssc unregistered [ 7019.351716] LNet: 178378:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7019.361977] LNetError: 178378:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7019.381603] LNet: Removed LNI 192.168.201.120@tcp [ 7020.385216] Key type .llcrypt unregistered [ 7020.389120] Key type ._llcrypt unregistered [ 7039.079896] Key type ._llcrypt registered [ 7039.083347] Key type .llcrypt registered [ 7039.294388] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 7040.403494] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7040.557931] alg: No test for adler32 (adler32-zlib) [ 7041.642296] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 7041.928787] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 7043.611761] Key type lgssc registered [ 7045.032801] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7052.538809] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7052.551723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7068.063652] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7073.183318] LustreError: 179623:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.120@tcp: failed processing log, type 4: rc = -110 [ 7078.303440] LustreError: 179623:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7078.309297] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7078.312122] LustreError: MGC192.168.201.120@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7078.319384] LustreError: 179623:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7078.322235] LustreError: 179623:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7078.326338] LustreError: 179623:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7078.365639] Lustre: server umount lustre-OST0000 complete [ 7078.367320] LustreError: 179623:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7087.922591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7089.323254] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7094.247208] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7099.875673] LustreError: 180121:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7099.887506] LustreError: 180121:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7100.909142] LustreError: 180122:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7103.124136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7103.530342] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7108.758226] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7116.041619] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7121.935750] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7128.694814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7128.819120] LustreError: 181936:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7128.831306] Lustre: 181936:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.201.120@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7128.928491] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7135.186347] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7142.036843] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7144.416485] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7144.425280] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7144.440702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7145.444533] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7145.445425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7145.468843] Lustre: Skipped 1 previous similar message [ 7145.488922] Lustre: Skipped 2 previous similar messages [ 7148.583132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7148.596021] Lustre: Skipped 2 previous similar messages [ 7153.634691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7153.637465] Lustre: Skipped 2 previous similar messages [ 7158.239131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7158.490858] Lustre: server umount lustre-MDT0000 complete [ 7158.752850] LustreError: 180122:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7158.766374] LustreError: 180122:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7162.063818] LustreError: 180101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785085048 with bad export cookie 3246373366398434021 [ 7162.079876] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7163.039428] LustreError: 180851:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000b4f4c127 x1871797212444672/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7163.208179] Lustre: server umount lustre-MDT0001 complete [ 7177.835638] Lustre: server umount lustre-OST0000 complete [ 7186.463310] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 7196.871053] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 7237.855596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7238.286657] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7238.324150] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7238.479272] Lustre: lustre-MDT0000: new disk, initializing [ 7238.590588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7238.609510] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7244.094754] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7255.440255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7255.614523] Lustre: 185464:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7255.664434] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7255.669178] Lustre: Skipped 1 previous similar message [ 7255.785792] Lustre: lustre-MDT0001: new disk, initializing [ 7255.948482] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7256.004455] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7256.023397] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7261.690560] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7274.274991] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7281.359685] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7288.805864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7289.035494] Lustre: lustre-OST0000: new disk, initializing [ 7289.038773] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7289.044022] Lustre: 187433:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7289.137482] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7295.066654] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7295.088659] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7295.201084] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7296.405198] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7307.405424] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7312.559552] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7312.758764] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7317.182676] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7317.424311] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7326.184558] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7326.187546] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7326.193684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7326.225762] Lustre: Skipped 1 previous similar message [ 7327.711644] Lustre: server umount lustre-OST0000 complete [ 7331.299679] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7336.416032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7336.418540] Lustre: Skipped 4 previous similar messages [ 7337.159942] Lustre: server umount lustre-MDT0000 complete [ 7340.952505] LustreError: 185454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785085226 with bad export cookie 3246373366398434742 [ 7340.959915] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7340.981481] LustreError: 185454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 7341.701507] Lustre: server umount lustre-MDT0001 complete [ 7352.936750] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 13:00:37 (1785085237) [ 7378.206925] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 7381.607944] Key type lgssc unregistered [ 7382.090747] LNet: 190393:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7382.103501] LNetError: 190393:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7382.121886] LNet: Removed LNI 192.168.201.120@tcp [ 7383.324888] Key type .llcrypt unregistered [ 7383.330667] Key type ._llcrypt unregistered [ 7404.325775] Key type ._llcrypt registered [ 7404.332962] Key type .llcrypt registered [ 7404.449869] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 7405.849835] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7405.977756] alg: No test for adler32 (adler32-zlib) [ 7407.116574] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 7407.445996] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 7409.216383] Key type lgssc registered [ 7410.737044] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7419.934469] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7419.955537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7421.419475] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7427.145768] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7431.650159] LustreError: 191613:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7431.668634] LustreError: 191613:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7436.662104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7436.776365] LustreError: 191613:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7437.174660] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7442.290418] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7450.933570] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7456.786577] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7466.597990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7466.763611] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7466.766257] LustreError: 193481:0:(llog.c:1696:llog_backup()) MGC192.168.201.120@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7466.778030] Lustre: 193481:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.201.120@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7466.906710] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7473.045954] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7483.035744] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7487.460251] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7487.462464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7487.465792] Lustre: Skipped 2 previous similar messages [ 7487.476169] Lustre: Skipped 2 previous similar messages [ 7492.584202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7492.592436] Lustre: Skipped 2 previous similar messages [ 7497.723404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7500.259868] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7500.589344] Lustre: server umount lustre-MDT0000 complete [ 7502.819975] LustreError: 192341:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7502.847948] LustreError: 192341:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7505.241180] LustreError: 191595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785085391 with bad export cookie 2024386530429833872 [ 7505.251404] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7505.605259] Lustre: server umount lustre-MDT0001 complete [ 7520.847538] Lustre: server umount lustre-OST0000 complete [ 7532.691260] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 7545.954124] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 7585.907844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7586.248264] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7586.300592] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7586.386859] Lustre: lustre-MDT0000: new disk, initializing [ 7586.505766] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7586.526765] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7592.558896] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7605.104391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7605.252864] Lustre: 197006:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7605.294119] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7605.295926] Lustre: Skipped 1 previous similar message [ 7605.298558] LustreError: 197788:0:(llog.c:1696:llog_backup()) MGC192.168.201.120@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7605.303515] LustreError: 197788:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7605.307269] Lustre: 197788:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.120@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7605.323448] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7605.326904] Lustre: Skipped 1 previous similar message [ 7605.511722] Lustre: lustre-MDT0001: new disk, initializing [ 7605.750370] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7605.845034] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7605.873390] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7611.204444] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7620.950776] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7627.309903] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7636.272530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7636.668600] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7636.671089] Lustre: Skipped 1 previous similar message [ 7636.677851] LustreError: 198952:0:(llog.c:1696:llog_backup()) MGC192.168.201.120@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7636.690691] LustreError: 198952:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7636.698343] Lustre: 198952:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.120@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7636.710707] Lustre: 198952:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7636.746743] Lustre: lustre-OST0000: new disk, initializing [ 7636.763376] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7636.786793] Lustre: 198976:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7636.940256] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7643.778101] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7643.790664] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7643.954321] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7644.431215] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7655.166677] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7660.656541] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7660.878099] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7666.025924] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7666.185416] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7669.728608] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7669.732209] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7669.737477] LustreError: Skipped 1 previous similar message [ 7669.739531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7669.753590] Lustre: Skipped 1 previous similar message [ 7669.783255] Lustre: Skipped 3 previous similar messages [ 7674.566658] Lustre: server umount lustre-OST0000 complete [ 7679.969904] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7679.978266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7679.988094] Lustre: Skipped 1 previous similar message [ 7680.008943] Lustre: Skipped 1 previous similar message [ 7685.887155] Lustre: server umount lustre-MDT0000 complete [ 7689.186797] LustreError: 197011:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7689.204315] LustreError: 197011:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7690.387250] LustreError: 196999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785085576 with bad export cookie 2024386530429834593 [ 7690.390266] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7690.405000] LustreError: 196999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 7690.993523] Lustre: server umount lustre-MDT0001 complete [ 7701.307150] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 13:06:26 (1785085586) [ 7719.230574] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 7733.312441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7733.970587] LustreError: 202105:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7734.042874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7740.897314] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7744.481306] LustreError: 202106:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7744.485770] LustreError: 202106:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7750.394418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7757.311523] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7760.968979] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7768.375326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7768.744650] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7768.751307] Lustre: Skipped 1 previous similar message [ 7776.112598] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7785.737693] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7786.014499] Lustre: lustre-OST0001: new disk, initializing [ 7786.018547] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7786.024447] Lustre: 204415:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7793.989709] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7794.836852] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7794.853742] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7794.980792] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7805.499268] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7811.435986] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7813.519194] Lustre: DEBUG MARKER: ADD OST3 [ 7818.583877] loop: module loaded [ 7818.664747] loop0: detected capacity change from 0 to 204800000 [ 7818.901060] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7818.925753] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7818.943750] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7826.033695] loop0: detected capacity change from 0 to 204800000 [ 7826.434758] Lustre: DEBUG MARKER: STOP OST3 [ 7832.037340] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7832.218564] Lustre: lustre-OST0003: new disk, initializing [ 7832.221353] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7832.234667] Lustre: 206459:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7834.013353] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7834.120460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7834.129334] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7834.184143] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7835.347202] Lustre: Failing over lustre-OST0003 [ 7835.604960] Lustre: server umount lustre-OST0003 complete [ 7836.077051] LustreError: 206609:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7838.669034] LustreError: 204810:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.201.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7838.693467] LustreError: 204810:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7839.236524] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7839.243378] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7839.810563] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7848.684136] Lustre: DEBUG MARKER: START OST3 again [ 7855.338756] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7855.799219] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7855.807236] Lustre: Skipped 2 previous similar messages [ 7855.816302] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7857.577408] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7857.738578] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 7857.740555] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7863.021317] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7884.558792] Lustre: Failing over lustre-OST0003 [ 7884.667798] Lustre: server umount lustre-OST0003 complete [ 7886.823612] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7886.855761] Lustre: Skipped 1 previous similar message [ 7886.874651] LustreError: 203589:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7886.899042] LustreError: 203589:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 7904.742241] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7904.745289] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7904.782166] Lustre: Skipped 1 previous similar message [ 7904.794479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7904.797748] Lustre: Skipped 2 previous similar messages [ 7910.070323] Lustre: server umount lustre-MDT0000 complete [ 7914.101046] LustreError: 203613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785085800 with bad export cookie 2024386530429835475 [ 7914.111323] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7914.115852] LustreError: 203613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7920.755197] Lustre: server umount lustre-MDT0001 complete [ 7925.089706] Lustre: server umount lustre-OST0000 complete [ 7929.467657] Lustre: server umount lustre-OST0001 complete [ 7936.990562] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 7947.638841] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 7992.667727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7992.986029] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7993.022875] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7993.114423] Lustre: lustre-MDT0000: new disk, initializing [ 7993.217166] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7993.252403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7998.193266] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8010.639160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8010.892361] Lustre: 211882:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8010.981636] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8010.987396] Lustre: Skipped 1 previous similar message [ 8011.117248] Lustre: lustre-MDT0001: new disk, initializing [ 8011.286970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8011.298820] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8016.367738] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8027.456861] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8033.854588] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8042.091165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8042.524370] Lustre: 213852:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8044.599183] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8044.625385] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8044.818056] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8050.608970] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8060.403771] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8066.143650] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8066.337606] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8072.150507] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8072.404638] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8075.749085] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8075.750116] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8075.760383] LustreError: Skipped 1 previous similar message [ 8075.761441] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8075.793910] Lustre: Skipped 6 previous similar messages [ 8075.825200] Lustre: Skipped 9 previous similar messages [ 8089.055119] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8089.377690] Lustre: server umount lustre-OST0000 complete [ 8093.156704] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8099.429748] Lustre: server umount lustre-MDT0000 complete [ 8101.347482] LustreError: 212684:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8101.374759] LustreError: 212684:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 17 previous similar messages [ 8103.720587] LustreError: 211877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785085989 with bad export cookie 2024386530429837771 [ 8103.722758] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8103.738709] LustreError: 211877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8103.995328] Lustre: server umount lustre-MDT0001 complete [ 8107.540535] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 13:13:12 (1785085992) [ 8125.973846] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 8137.969505] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 8174.521585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8174.918621] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8174.960678] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8174.969226] Lustre: Skipped 1 previous similar message [ 8175.087398] Lustre: lustre-MDT0000: new disk, initializing [ 8175.096200] Lustre: Skipped 1 previous similar message [ 8175.230522] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8180.697671] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8191.454779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8191.624193] Lustre: 218466:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8192.014102] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8197.700586] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8207.885770] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8213.507343] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8219.658982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8219.935501] Lustre: 220440:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8221.649236] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8226.116600] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8235.807566] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8240.190162] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8240.517481] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8245.339488] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8245.600985] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8252.390997] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8252.393135] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8252.397673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8252.397691] Lustre: Skipped 7 previous similar messages [ 8252.417410] Lustre: Skipped 3 previous similar messages [ 8253.745565] Lustre: server umount lustre-OST0000 complete [ 8263.415411] Lustre: server umount lustre-MDT0000 complete [ 8266.977188] LustreError: 218457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785086153 with bad export cookie 2024386530429838653 [ 8266.981823] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8266.989459] LustreError: 218457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8267.599894] Lustre: server umount lustre-MDT0001 complete [ 8278.816150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8279.497275] LustreError: 222658:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8279.583533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8279.587683] Lustre: Skipped 5 previous similar messages [ 8286.769500] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8295.317368] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8301.003942] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8308.801431] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8315.265691] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8322.475532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8330.376675] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8341.188903] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8354.793230] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 8354.800453] LustreError: Skipped 1 previous similar message [ 8354.807471] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8354.827279] Lustre: Skipped 2 previous similar messages [ 8354.833074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8354.835197] Lustre: Skipped 5 previous similar messages [ 8357.987964] Lustre: server umount lustre-OST0000 complete [ 8371.106784] Lustre: server umount lustre-MDT0000 complete [ 8375.000219] LustreError: 222638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785086261 with bad export cookie 2024386530429839535 [ 8375.009922] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8375.025471] LustreError: 222638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8375.383969] Lustre: server umount lustre-MDT0001 complete [ 8483.034874] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 8485.912987] Key type lgssc unregistered [ 8486.161799] LNet: 226322:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8486.168789] LNetError: 226322:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8486.181954] LNet: Removed LNI 192.168.201.120@tcp [ 8486.997242] Key type .llcrypt unregistered [ 8486.998472] Key type ._llcrypt unregistered [ 8512.799707] Key type ._llcrypt registered [ 8512.802547] Key type .llcrypt registered [ 8512.977819] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 8530.858581] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 8531.954374] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8532.009818] alg: No test for adler32 (adler32-zlib) [ 8533.403310] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 8533.637393] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 8535.314518] Key type lgssc registered [ 8536.711203] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8578.047820] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8578.069503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8579.379149] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8579.442210] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8579.606123] Lustre: lustre-MDT0000: new disk, initializing [ 8579.750150] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8579.772320] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8585.861331] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8596.938113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8597.194246] Lustre: 229679:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8597.323440] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8597.327175] Lustre: Skipped 1 previous similar message [ 8597.432573] Lustre: lustre-MDT0001: new disk, initializing [ 8597.555202] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8597.616240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8597.635471] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8603.450298] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8613.772965] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8619.229653] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8626.011658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8626.332537] Lustre: lustre-OST0000: new disk, initializing [ 8626.336793] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8626.342912] Lustre: 231644:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8626.497903] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8632.660832] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8632.946691] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8632.971538] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8633.136441] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8642.321900] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8646.934149] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8647.120546] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8651.391651] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8651.645083] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8653.792924] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8653.797983] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8653.812446] Lustre: Skipped 1 previous similar message [ 8653.813228] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8653.837470] Lustre: Skipped 1 previous similar message [ 8658.945594] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8658.960722] Lustre: Skipped 1 previous similar message [ 8659.525191] Lustre: server umount lustre-OST0000 complete [ 8664.038330] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8664.054069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8664.060699] Lustre: Skipped 1 previous similar message [ 8669.152722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8669.155263] Lustre: Skipped 1 previous similar message [ 8669.671671] Lustre: server umount lustre-MDT0000 complete [ 8672.984230] LustreError: 229668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785086559 with bad export cookie 14133213247949726284 [ 8672.988366] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8672.991126] LustreError: 229668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8673.288692] Lustre: server umount lustre-MDT0001 complete [ 8692.798927] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 8703.576954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8704.012036] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8709.088358] LustreError: 234578:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8709.119776] LustreError: 234578:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8709.326479] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8713.189119] LustreError: 234579:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8718.312760] LustreError: 234578:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8719.043547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8719.509351] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8724.070983] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8727.152392] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8733.628883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8734.037528] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8740.126978] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8748.580108] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8748.805770] Lustre: lustre-OST0001: new disk, initializing [ 8748.809210] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8748.814425] Lustre: 236886:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8755.651928] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8758.892064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8758.901244] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8758.957071] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8766.285879] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8774.434803] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8787.048710] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 13:24:31 (1785086671) [ 8789.989719] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8789.994880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8790.006447] Lustre: Skipped 4 previous similar messages [ 8790.011916] Lustre: Skipped 2 previous similar messages [ 8795.718068] Lustre: server umount lustre-MDT0000 complete [ 8799.199045] LustreError: 234560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785086685 with bad export cookie 14133213247949727166 [ 8799.200250] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8799.218922] LustreError: 234560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8799.636562] Lustre: server umount lustre-MDT0001 complete [ 8813.902179] Lustre: server umount lustre-OST0000 complete [ 8816.097257] Lustre: 227902:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785086686/real 1785086686] req@ffff890d5273ad80 x1871798776996352/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785086702 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8816.141622] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8817.863585] Lustre: server umount lustre-OST0001 complete [ 8824.147245] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 8834.694405] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 8874.101813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8874.359258] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8874.381139] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8874.442103] Lustre: lustre-MDT0000: new disk, initializing [ 8874.516761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8874.523117] Lustre: Skipped 1 previous similar message [ 8874.535575] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8879.500442] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8892.525060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8892.669987] Lustre: 240830:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8892.724153] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8892.728161] Lustre: Skipped 1 previous similar message [ 8892.819397] Lustre: lustre-MDT0001: new disk, initializing [ 8892.990660] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8893.002375] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8898.908884] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8910.466412] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8917.466068] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8924.951076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8925.240348] Lustre: lustre-OST0000: new disk, initializing [ 8925.243476] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8925.248778] Lustre: 242798:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8925.348467] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8925.353531] Lustre: Skipped 1 previous similar message [ 8926.678236] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8926.685723] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8926.805379] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8932.561487] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8942.226193] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8947.331497] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8947.600859] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8953.761983] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8954.065485] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8957.413517] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8957.418248] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8957.441240] LustreError: Skipped 2 previous similar messages [ 8957.461368] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8957.475190] Lustre: Skipped 1 previous similar message [ 8957.495742] Lustre: Skipped 6 previous similar messages [ 8962.584620] Lustre: server umount lustre-OST0000 complete [ 8967.667661] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8967.677553] Lustre: Skipped 1 previous similar message [ 8973.017179] Lustre: server umount lustre-MDT0000 complete [ 8977.883524] LustreError: 240821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785086863 with bad export cookie 14133213247949729301 [ 8977.888739] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8977.895887] LustreError: 240821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8978.384743] Lustre: server umount lustre-MDT0001 complete [ 8988.834260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8989.366373] LustreError: 245016:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8989.485897] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8994.481446] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8994.785508] LustreError: 245017:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9002.414866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9007.093504] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 9014.153077] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9021.672974] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9029.272557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9036.674766] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 9045.157711] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9191.394310] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9191.406371] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9191.427729] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9191.427737] Lustre: Skipped 4 previous similar messages [ 9196.539868] Lustre: server umount lustre-OST0000 complete [ 9207.048203] Lustre: server umount lustre-MDT0000 complete [ 9211.247267] LustreError: 244997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785087097 with bad export cookie 14133213247949730183 [ 9211.251309] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9211.254051] LustreError: 244997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9211.663809] Lustre: server umount lustre-MDT0001 complete [ 9316.059174] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 9319.650283] Key type lgssc unregistered [ 9320.026775] LNet: 248628:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9320.033085] LNetError: 248628:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9320.052929] LNet: Removed LNI 192.168.201.120@tcp [ 9321.278206] Key type .llcrypt unregistered [ 9321.281111] Key type ._llcrypt unregistered [ 9352.324221] Key type ._llcrypt registered [ 9352.331855] Key type .llcrypt registered [ 9352.510371] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 9369.897110] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 9371.339856] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9371.451499] alg: No test for adler32 (adler32-zlib) [ 9372.742479] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 9372.971293] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 9374.663875] Key type lgssc registered [ 9376.164578] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9431.845149] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 9445.800375] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9445.843941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9447.186383] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9447.212501] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9447.364815] Lustre: lustre-MDT0000: new disk, initializing [ 9447.555535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9447.607830] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9453.224399] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 9465.268056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9465.392533] Lustre: 252722:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9465.446069] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9465.449194] Lustre: Skipped 1 previous similar message [ 9465.518385] Lustre: lustre-MDT0001: new disk, initializing [ 9465.691268] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9465.747489] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9465.759596] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9471.610625] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 9476.971791] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9484.340218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9484.613382] Lustre: lustre-OST0000: new disk, initializing [ 9484.618281] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9484.625663] Lustre: 254357:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9484.738297] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9491.017043] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9491.026259] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9491.170523] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9491.589478] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 9503.323625] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9503.641579] Lustre: lustre-OST0001: new disk, initializing [ 9503.667972] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9503.703886] Lustre: 255226:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9503.898359] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9509.428742] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9509.433742] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9509.499452] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9510.952910] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 9523.123072] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9532.992166] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9550.307413] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9550.317929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9550.346045] Lustre: Skipped 2 previous similar messages [ 9550.346872] Lustre: Skipped 1 previous similar message [ 9554.358543] Lustre: server umount lustre-MDT0000 complete [ 9558.746508] LustreError: 252715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785087444 with bad export cookie 18147961116336882182 [ 9558.768957] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9559.543106] Lustre: server umount lustre-MDT0001 complete [ 9575.008188] Lustre: server umount lustre-OST0000 complete [ 9576.735162] Lustre: 250217:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785087446/real 1785087446] req@ffff890d7eeeb100 x1871799656726912/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785087462 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9576.759544] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9576.777173] Lustre: Skipped 1 previous similar message [ 9579.131305] Lustre: server umount lustre-OST0001 complete [ 9595.587351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9606.170549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9613.715196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9622.293094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9644.508505] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 9653.923571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9653.951902] Lustre: lustre-MDT0000: reset Object Index mappings [ 9654.468270] LustreError: 260238:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9654.484947] LustreError: 260238:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 9654.562321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9659.412895] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 9659.877415] LustreError: 260239:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9665.004786] LustreError: 260238:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9668.025785] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9668.042489] Lustre: lustre-MDT0001: reset Object Index mappings [ 9668.502445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9673.157747] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 9675.855736] Lustre: 261382:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9682.548066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9689.933179] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 9693.167491] LustreError: 261736:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9698.288994] LustreError: 261734:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9698.530736] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9698.899814] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9698.902303] Lustre: Skipped 1 previous similar message [ 9700.991304] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9701.019839] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9706.742557] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 9715.054273] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9718.992880] Lustre: 263253:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9724.901140] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9724.915186] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9724.939957] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9724.945920] Lustre: Skipped 2 previous similar messages [ 9726.050835] Lustre: server umount lustre-MDT0000 complete [ 9726.954549] LustreError: 260972:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9726.957474] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9726.970714] LustreError: 260972:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9727.003941] Lustre: Skipped 2 previous similar messages [ 9729.515707] LustreError: 263255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785087615 with bad export cookie 18147961116336888972 [ 9729.520936] LustreError: 263255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9729.533965] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9729.942337] Lustre: server umount lustre-MDT0001 complete [ 9744.325939] Lustre: server umount lustre-OST0000 complete [ 9748.449985] Lustre: 250216:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785087618/real 1785087618] req@ffff890d77e28380 x1871799656775680/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785087634 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9748.487725] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9748.826589] Lustre: server umount lustre-OST0001 complete [ 9768.354022] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 13:40:52 (1785087652) [ 9783.821116] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 9793.727587] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 9827.368266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9827.601442] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9827.628158] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9827.693470] Lustre: lustre-MDT0000: new disk, initializing [ 9827.752815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9827.771147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9831.701238] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 9841.652435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9841.811516] Lustre: 267507:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9841.867666] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9841.871452] Lustre: Skipped 1 previous similar message [ 9841.959087] Lustre: lustre-MDT0001: new disk, initializing [ 9842.157357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9842.166995] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9846.288653] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 9854.656723] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9859.993870] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9866.359689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9866.647405] Lustre: lustre-OST0000: new disk, initializing [ 9866.651288] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9866.660161] Lustre: 269475:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9868.154310] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9868.164496] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9868.301206] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9873.136366] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 9882.604801] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9887.042936] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9887.149488] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9891.436917] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9891.599432] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9893.860357] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9893.862304] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9893.871394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9893.883291] Lustre: Skipped 1 previous similar message [ 9893.894223] Lustre: Skipped 1 previous similar message [ 9898.981366] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9898.993385] Lustre: Skipped 1 previous similar message [ 9899.631615] Lustre: server umount lustre-OST0000 complete [ 9908.704901] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9908.714070] LustreError: Skipped 1 previous similar message [ 9908.740449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9910.899894] Lustre: server umount lustre-MDT0000 complete [ 9914.340040] LustreError: 267519:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9914.368486] LustreError: 267519:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9915.113662] LustreError: 267499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785087801 with bad export cookie 18147961116336891058 [ 9915.124683] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9915.128841] LustreError: 267499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9915.510388] Lustre: server umount lustre-MDT0001 complete [ 9935.798511] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 9945.864273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9946.345666] LustreError: 272410:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9946.395209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9946.397406] Lustre: Skipped 2 previous similar messages [ 9951.064379] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 9958.398735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9962.431409] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 9964.831180] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9970.235184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9977.824498] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 9986.891684] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9987.088713] Lustre: lustre-OST0001: new disk, initializing [ 9987.091876] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9987.101228] Lustre: 274718:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9988.948978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9988.964287] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9989.101294] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9993.930404] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [10003.547204] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10006.750318] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10012.110989] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10012.449191] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10020.350310] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10020.368611] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10020.394756] Lustre: Skipped 2 previous similar messages [10020.396561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10020.407807] Lustre: Skipped 2 previous similar messages [10026.697204] Lustre: server umount lustre-MDT0000 complete [10030.064652] LustreError: 272407:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10030.094901] LustreError: 272407:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10030.846874] LustreError: 272392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785087916 with bad export cookie 18147961116336891947 [10030.850728] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10030.870664] LustreError: 272392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10031.376867] Lustre: server umount lustre-MDT0001 complete [10044.439208] Lustre: server umount lustre-OST0000 complete [10058.982363] Lustre: server umount lustre-OST0001 complete [10074.562850] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [10087.490830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10088.161524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10088.173804] Lustre: Skipped 3 previous similar messages [10092.837957] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [10102.435185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10108.509674] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [10111.460740] Lustre: 278914:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10117.829344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10124.670679] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [10133.505496] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10133.969421] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10139.673556] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10141.954671] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [10149.284496] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10153.190146] Lustre: 280793:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10158.074422] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10158.224229] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10168.259179] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 13:47:32 (1785088052) [10184.674931] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10184.681939] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10184.691341] Lustre: Skipped 3 previous similar messages [10184.696268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10184.700224] Lustre: Skipped 4 previous similar messages [10186.499215] Lustre: server umount lustre-MDT0000 complete [10189.851122] LustreError: 277754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785088075 with bad export cookie 18147961116336898499 [10189.873992] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10190.822768] LustreError: 280656:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10190.853653] LustreError: 280656:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [10196.522744] Lustre: server umount lustre-MDT0001 complete [10199.977919] LustreError: 250215:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff890d4e7d8380 x1871799656963072/t0(0) o38->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10214.368321] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10214.500372] Lustre: server umount lustre-OST0000 complete [10218.419033] Lustre: server umount lustre-OST0001 complete [10225.401569] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [10236.273854] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [10293.057040] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [10306.666357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10307.023303] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10307.053503] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10307.136910] Lustre: lustre-MDT0000: new disk, initializing [10307.188040] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10312.468068] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [10323.268893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10323.393462] Lustre: 285058:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10323.671558] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10328.943800] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [10342.009684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10342.452966] Lustre: 286686:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10344.105217] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10350.232928] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [10361.463804] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10361.774062] Lustre: 287560:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10361.903770] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10361.912199] Lustre: Skipped 6 previous similar messages [10363.588797] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10363.599470] Lustre: Skipped 1 previous similar message [10363.789450] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10368.644608] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [10378.701544] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10382.825382] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10382.828586] Lustre: Skipped 1 previous similar message [10400.281451] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 13:51:24 (1785088284) [10404.896100] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10404.909957] Lustre: Skipped 5 previous similar messages [10404.990980] Lustre: server umount lustre-MDT0000 complete [10408.869447] LustreError: 285048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785088294 with bad export cookie 18147961116336906129 [10408.883929] LustreError: 285048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10408.889487] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10409.290325] Lustre: server umount lustre-MDT0001 complete [10424.551984] Lustre: server umount lustre-OST0000 complete [10425.119090] Lustre: 250216:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785088295/real 1785088295] req@ffff890d44af2680 x1871799657036544/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785088311 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10428.660868] Lustre: server umount lustre-OST0001 complete [10437.343790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10442.807774] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [10447.335772] LustreError: 289804:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10447.362462] LustreError: 289804:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [10451.818980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10456.967311] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [10464.972319] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10471.280474] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10479.482214] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10487.278618] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [10496.306421] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10516.291863] loop1: detected capacity change from 0 to 204800000 [10516.592747] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10516.616102] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10516.639837] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10522.488782] loop1: detected capacity change from 0 to 204800000 [10528.888718] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10529.031154] LustreError: 289792:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs29734: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10529.052707] LustreError: 289792:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs29734 [10529.065783] LustreError: 289792:0:(mgs_handler.c:517:mgs_target_reg()) fs29734-MDT0001: Error checking target index: rc = -13 [10529.083224] LustreError: fs29734-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10529.090363] LustreError: 293525:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10529.096588] LustreError: 293525:0:(tgt_mount.c:2129:server_put_super()) no obd fs29734-MDT0001 [10529.100730] LustreError: 293525:0:(tgt_mount.c:116:server_deregister_mount()) fs29734-MDT0001 not registered [10529.132347] Lustre: server umount fs29734-MDT0001 complete [10529.134988] LustreError: 293525:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [10537.497824] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10537.576607] Lustre: Setting parameter fs29734-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs29734-MDT0001 [10537.637643] Lustre: srv-fs29734-MDT0001: No data found on store. Initialize space. [10537.640979] Lustre: Skipped 4 previous similar messages [10537.713188] Lustre: fs29734-MDT0001: new disk, initializing [10537.718564] Lustre: Skipped 3 previous similar messages [10537.921247] Lustre: srv-fs29734-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10537.937258] Lustre: 293991:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs29734-MDT0001-mdtlov: can't create new llog: rc = -115 [10537.948276] LustreError: 293991:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs29734-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10542.373713] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [10549.030939] Lustre: server umount fs29734-MDT0001 complete [10561.479571] loop1: detected capacity change from 0 to 204800000 [10564.083180] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10567.427449] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10567.650782] Lustre: 295557:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs29734-MDT0001-mdtlov: can't create new llog: rc = -115 [10567.663247] LustreError: 295557:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs29734-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10573.050274] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [10577.765648] Lustre: server umount fs29734-MDT0001 complete [10589.105275] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.20@tcp (stopping) [10589.111726] Lustre: Skipped 8 previous similar messages [10589.668188] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10589.688822] Lustre: Skipped 6 previous similar messages [10595.277271] Lustre: server umount lustre-MDT0000 complete [10603.589983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10603.726243] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10609.142780] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10609.358397] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [10617.272071] loop1: detected capacity change from 0 to 204800000 [10622.773658] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10623.441971] Lustre: 298006:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs29734-MDT0001-mdtlov: can't create new llog: rc = -115 [10623.445537] LustreError: 298006:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs29734-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10629.358158] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [10634.283307] Lustre: server umount fs29734-MDT0001 complete [10660.337558] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10660.350962] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10660.360576] LustreError: Skipped 1 previous similar message [10660.365048] Lustre: Skipped 7 previous similar messages [10664.711438] Lustre: server umount lustre-OST0000 complete [10675.658891] Lustre: server umount lustre-MDT0000 complete [10679.759353] LustreError: 289785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785088565 with bad export cookie 18147961116336908810 [10679.768796] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10679.772530] LustreError: 289785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [10680.079571] Lustre: server umount lustre-MDT0001 complete [10693.574859] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [10696.776216] Key type lgssc unregistered [10697.068387] LNet: 300350:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10697.071790] LNetError: 300350:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10697.087215] LNet: Removed LNI 192.168.201.120@tcp [10697.900143] Key type .llcrypt unregistered [10697.901453] Key type ._llcrypt unregistered [10720.421323] Key type ._llcrypt registered [10720.424717] Key type .llcrypt registered [10720.566701] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [10721.985606] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10722.116063] alg: No test for adler32 (adler32-zlib) [10723.384727] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [10723.580494] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [10725.255878] Key type lgssc registered [10726.784192] Lustre: Echo OBD driver; http://www.lustre.org/ [10734.533449] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10734.564819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10736.165938] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10741.473104] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [10746.337896] LustreError: 301624:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10746.362483] LustreError: 301624:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10750.753539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10751.287142] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10756.454592] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [10764.357968] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10771.117469] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10779.176885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10779.619787] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10781.673846] LustreError: 303580:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10786.800351] LustreError: 303454:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10787.621637] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [10791.926246] LustreError: 303454:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10796.021878] LustreError: 303452:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10796.060290] LustreError: 303452:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10796.845512] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10806.263132] LustreError: 303580:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10806.286872] LustreError: 303580:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10815.360365] loop1: detected capacity change from 0 to 204800000 [10815.511899] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10815.523776] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10815.542856] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10821.651983] loop1: detected capacity change from 0 to 204800000 [10824.091740] LustreError: 303454:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10824.128282] LustreError: 303454:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [10830.377384] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10830.620504] Lustre: fs11728-OST0000: new disk, initializing [10830.625237] Lustre: srv-fs11728-OST0000: No data found on store. Initialize space. [10830.632786] Lustre: 305361:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10830.754725] Lustre: fs11728-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10839.165484] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [10848.408821] Lustre: server umount fs11728-OST0000 complete [10857.451720] LustreError: 303455:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10857.484517] LustreError: 303455:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 18 previous similar messages [10862.576109] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10862.677978] loop1: detected capacity change from 0 to 204800000 [10869.602169] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10870.007325] Lustre: fs11728-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10877.961308] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10878.821116] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [10890.149323] loop2: detected capacity change from 0 to 204800000 [10890.490364] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10890.517594] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10890.550196] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10896.359651] loop2: detected capacity change from 0 to 204800000 [10902.721567] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10902.901577] LustreError: fs11728-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [10902.910408] LustreError: fs11728-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10902.922023] LustreError: 308401:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10902.930434] LustreError: 308401:0:(tgt_mount.c:2129:server_put_super()) no obd fs11728-OST0001 [10902.936277] LustreError: 308401:0:(tgt_mount.c:116:server_deregister_mount()) fs11728-OST0001 not registered [10902.966388] Lustre: server umount fs11728-OST0001 complete [10902.967808] LustreError: 308401:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [10914.730469] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10914.913355] Lustre: fs11728-OST0001: new disk, initializing [10914.917488] Lustre: srv-fs11728-OST0001: No data found on store. Initialize space. [10914.924318] Lustre: 308954:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10915.013416] Lustre: fs11728-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10922.202473] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [10924.004161] LustreError: 303580:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10924.012094] LustreError: 303580:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 39 previous similar messages [10931.530324] Lustre: server umount fs11728-OST0000 complete [10942.335788] Lustre: server umount fs11728-OST0001 complete [10956.143576] loop1: detected capacity change from 0 to 204800000 [10961.862786] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10962.097962] Lustre: fs11728-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10969.456195] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [10977.065502] loop2: detected capacity change from 0 to 204800000 [10983.801392] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10992.670673] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [11002.708369] Lustre: server umount fs11728-OST0000 complete [11012.489491] Lustre: server umount fs11728-OST0001 complete [11016.189363] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11031.525208] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11031.527908] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11031.530416] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11031.544642] Lustre: Skipped 1 previous similar message [11034.963392] Lustre: server umount lustre-OST0000 complete [11041.761359] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11041.765437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11041.775733] Lustre: Skipped 1 previous similar message [11041.788591] Lustre: Skipped 2 previous similar messages [11045.742317] Lustre: server umount lustre-MDT0000 complete [11049.979782] LustreError: 301606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785088936 with bad export cookie 15830266097319400777 [11049.992782] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11049.996177] LustreError: 301606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11050.391808] Lustre: server umount lustre-MDT0001 complete [11064.588573] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [11068.208926] Key type lgssc unregistered [11068.558772] LNet: 314894:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11068.571317] LNetError: 314894:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11068.601910] LNet: Removed LNI 192.168.201.120@tcp [11069.792191] Key type .llcrypt unregistered [11069.793719] Key type ._llcrypt unregistered [11104.804557] Key type ._llcrypt registered [11104.810338] Key type .llcrypt registered [11104.957341] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [11123.632408] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [11124.654705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11124.668624] alg: No test for adler32 (adler32-zlib) [11125.846461] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [11126.079445] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [11127.751685] Key type lgssc registered [11128.934473] Lustre: Echo OBD driver; http://www.lustre.org/ [11167.168678] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11167.222157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11168.628631] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11168.667740] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11168.763571] Lustre: lustre-MDT0000: new disk, initializing [11168.901748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11168.946881] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11174.377737] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [11185.586024] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11185.736864] Lustre: 318753:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11185.797906] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11185.800689] Lustre: Skipped 1 previous similar message [11185.854841] Lustre: lustre-MDT0001: new disk, initializing [11185.935553] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11185.963721] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11185.977688] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11190.824552] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [11200.447437] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11206.743736] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11214.502027] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11214.970237] Lustre: lustre-OST0000: new disk, initializing [11214.984216] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11215.002528] Lustre: 320720:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11215.132806] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11222.420678] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [11223.170058] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11223.183781] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11223.335557] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11231.583929] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11233.944705] Lustre: Mounted lustre-client [11235.048032] Lustre: Unmounted lustre-client [11249.120660] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11249.123515] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11249.127935] LustreError: Skipped 1 previous similar message [11249.138281] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11249.141313] Lustre: Skipped 1 previous similar message [11249.163588] Lustre: Skipped 1 previous similar message [11251.451435] Lustre: server umount lustre-OST0000 complete [11257.825241] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11257.838574] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11257.850808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11259.873376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11259.875175] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11261.796510] Lustre: server umount lustre-MDT0000 complete [11265.536592] LustreError: 318743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785089151 with bad export cookie 4508959814236381349 [11265.538297] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11265.549307] LustreError: 318743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11266.009649] Lustre: server umount lustre-MDT0001 complete [11278.842796] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [11282.071791] Key type lgssc unregistered [11295.072079] LNet: 1 local NIs in recovery (showing 1): 192.168.201.120@tcp [11295.775842] LNet: 322882:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11295.789342] LNetError: 322882:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11363.367933] LNet: Removed LNI 192.168.201.120@tcp [11364.317708] Key type .llcrypt unregistered [11364.321288] Key type ._llcrypt unregistered [11385.687250] Key type ._llcrypt registered [11385.689686] Key type .llcrypt registered [11385.845545] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [11405.037913] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [11407.103283] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11407.162074] alg: No test for adler32 (adler32-zlib) [11408.344389] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [11408.653968] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [11410.319196] Key type lgssc registered [11411.730299] Lustre: Echo OBD driver; http://www.lustre.org/ [11447.906775] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 14:08:52 (1785089332) [11455.651257] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11455.706957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11457.054631] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11457.094747] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11457.214946] Lustre: lustre-MDT0000: new disk, initializing [11457.311952] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11457.334245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11463.180675] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [11474.746543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11474.932633] Lustre: 326108:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11474.992128] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11474.995822] Lustre: Skipped 1 previous similar message [11475.077660] Lustre: lustre-MDT0001: new disk, initializing [11475.179730] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11475.217147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11475.226436] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11481.151752] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [11491.387568] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11499.018588] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11506.966203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11507.492737] Lustre: lustre-OST0000: new disk, initializing [11507.500130] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11507.511110] Lustre: 328077:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11507.604558] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11514.045039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11514.063372] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11514.264743] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11515.186642] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [11525.641577] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11545.056637] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11545.064689] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11545.079047] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11545.111308] Lustre: Skipped 1 previous similar message [11546.229518] Lustre: server umount lustre-OST0000 complete [11552.231798] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11552.262709] LustreError: Skipped 1 previous similar message [11552.279279] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11552.307588] Lustre: Skipped 1 previous similar message [11552.314272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11555.300296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11555.302452] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11557.774505] LustreError: 329231:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x5:0x0].0x0 (000000007ee566d1) refcount nonzero (1) after lock cleanup; forcing cleanup. [11558.036218] Lustre: server umount lustre-MDT0000 complete [11564.806234] LustreError: 326101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785089450 with bad export cookie 10586507250382604010 [11564.814469] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11564.820842] LustreError: 326101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11565.458250] Lustre: server umount lustre-MDT0001 complete [11578.305576] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [11581.376708] Key type lgssc unregistered [11581.625559] LNet: 329982:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11581.629697] LNetError: 329982:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11581.648814] LNet: Removed LNI 192.168.201.120@tcp [11582.601694] Key type .llcrypt unregistered [11582.604991] Key type ._llcrypt unregistered [11604.782399] Key type ._llcrypt registered [11604.784391] Key type .llcrypt registered [11604.891881] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [11606.722730] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11606.803253] alg: No test for adler32 (adler32-zlib) [11607.908727] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [11608.203272] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [11609.928144] Key type lgssc registered [11611.573248] Lustre: Echo OBD driver; http://www.lustre.org/ [11619.582364] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11619.598602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11621.156976] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11625.515513] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [11631.586681] LustreError: 331258:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11631.598908] LustreError: 331258:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11634.016665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11634.438698] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11639.923416] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [11648.027953] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11656.823611] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11666.562281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11666.951800] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11671.966291] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [11680.768811] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11701.793078] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11703.275800] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11708.331797] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.20@tcp (stopping) [11711.457480] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11711.472631] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11711.490797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11711.973322] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11711.985345] Lustre: Skipped 1 previous similar message [11713.419200] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.20@tcp (stopping) [11713.441749] Lustre: Skipped 2 previous similar messages [11713.582853] Lustre: server umount lustre-MDT0000 complete [11717.100544] LustreError: 331257:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11717.127493] LustreError: 331257:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11717.479894] LustreError: 331240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785089603 with bad export cookie 4482770653853912069 [11717.505598] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11717.511788] LustreError: 331240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11718.535039] LustreError: 331254:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11718.599202] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.20@tcp (stopping) [11722.208571] LustreError: 331253:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11722.211253] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11723.672295] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.20@tcp (stopping) [11723.682869] Lustre: Skipped 1 previous similar message [11723.917053] Lustre: server umount lustre-MDT0001 complete [11727.307636] Lustre: server umount lustre-OST0000 complete [11736.470990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11741.186985] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [11744.189745] Lustre: 335466:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.120@tcp [11746.262418] Lustre: server umount MGS complete [11755.141654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11755.895773] LustreError: 336038:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11755.920533] LustreError: 336038:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11755.969822] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11760.549447] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [11765.223916] LustreError: 336039:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11765.246569] LustreError: 336039:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11769.571832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11770.006868] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11775.264741] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [11783.964127] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11784.281256] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11791.125816] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [11798.848093] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11800.499115] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11806.649750] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11808.427188] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11815.411952] Lustre: server umount lustre-OST0000 complete [11821.026813] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11821.040777] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11821.073704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11829.813667] Lustre: server umount lustre-MDT0000 complete [11831.776987] Lustre: 330569:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785089701/real 1785089701] req@00000000076fa842 x1871802001324288/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1785089717 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11831.781947] LustreError: 336033:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11831.813206] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11831.813215] Lustre: Skipped 1 previous similar message [11831.846968] LustreError: 336033:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11832.415219] Lustre: 330571:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785089702/real 1785089702] req@00000000e0dd63e2 x1871802001325184/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1785089718 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11833.710410] LustreError: 336020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785089719 with bad export cookie 4482770653853912909 [11833.718306] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11833.721227] LustreError: 336020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11834.106600] Lustre: server umount lustre-MDT0001 complete [11846.624282] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [11849.656504] Key type lgssc unregistered [11849.928731] LNet: 339473:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11849.942461] LNetError: 339473:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11849.968410] LNet: Removed LNI 192.168.201.120@tcp [11850.866790] Key type .llcrypt unregistered [11850.868476] Key type ._llcrypt unregistered [11873.644054] Key type ._llcrypt registered [11873.646153] Key type .llcrypt registered [11873.765470] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [11892.713362] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [11894.886195] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11894.943744] alg: No test for adler32 (adler32-zlib) [11896.033844] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [11896.247920] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [11897.879185] Key type lgssc registered [11899.466310] Lustre: Echo OBD driver; http://www.lustre.org/ [11939.712173] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11939.745389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11941.027777] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11941.067429] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11941.158069] Lustre: lustre-MDT0000: new disk, initializing [11941.271717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11941.314146] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11946.270423] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [11957.922196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11958.051447] Lustre: 342638:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11958.111567] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11958.114036] Lustre: Skipped 1 previous similar message [11958.211694] Lustre: lustre-MDT0001: new disk, initializing [11958.329102] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11958.393960] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11958.413412] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11963.644634] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [11973.533630] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11980.199373] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11988.054849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11988.255309] Lustre: lustre-OST0000: new disk, initializing [11988.258674] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11988.263452] Lustre: 344605:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11988.336142] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11995.182453] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [11998.251169] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11998.268769] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11998.373117] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12005.084051] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12010.078375] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12010.286555] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12015.065262] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12015.285576] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12019.168463] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12019.171594] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12019.179549] LustreError: Skipped 1 previous similar message [12019.188105] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12019.206703] Lustre: Skipped 1 previous similar message [12019.224356] Lustre: Skipped 1 previous similar message [12023.402119] Lustre: server umount lustre-OST0000 complete [12029.410950] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12029.433807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12034.024279] Lustre: server umount lustre-MDT0000 complete [12037.896145] LustreError: 342628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785089923 with bad export cookie 11165107441612612632 [12037.905104] LustreError: 342628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12037.906377] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12038.613265] Lustre: server umount lustre-MDT0001 complete [12049.140906] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 14:18:53 (1785089933) [12067.383393] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [12079.947110] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [12120.962632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12121.145608] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12121.176673] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12121.260483] Lustre: lustre-MDT0000: new disk, initializing [12121.339332] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12121.353567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12126.692028] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [12138.177768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12138.350633] Lustre: 349368:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12138.422599] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12138.425244] Lustre: Skipped 1 previous similar message [12138.513993] Lustre: lustre-MDT0001: new disk, initializing [12138.628230] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12138.680395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12138.704508] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12145.272352] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [12156.487326] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12163.093180] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12169.921452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12170.136951] Lustre: lustre-OST0000: new disk, initializing [12170.140619] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12170.145663] Lustre: 351338:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12170.218657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12177.049644] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [12177.468082] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12177.481187] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12177.592056] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12187.190535] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12223.970664] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12223.974325] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12223.989351] LustreError: Skipped 1 previous similar message [12224.005357] Lustre: Skipped 1 previous similar message [12224.014362] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12224.036034] Lustre: Skipped 2 previous similar messages [12229.096305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12229.105011] Lustre: Skipped 1 previous similar message [12229.783586] Lustre: server umount lustre-OST0000 complete [12234.211686] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12234.214874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12234.243179] Lustre: Skipped 2 previous similar messages [12234.272261] Lustre: Skipped 1 previous similar message [12240.207965] Lustre: server umount lustre-MDT0000 complete [12244.431096] LustreError: 350164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785090130 with bad export cookie 11165107441612613521 [12244.433649] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12244.440863] LustreError: 350164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12244.877638] Lustre: server umount lustre-MDT0001 complete [12258.699541] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [12261.942342] Key type lgssc unregistered [12262.286680] LNet: 353510:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12262.291907] LNetError: 353510:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12262.320771] LNet: Removed LNI 192.168.201.120@tcp [12263.359224] Key type .llcrypt unregistered [12263.362512] Key type ._llcrypt unregistered [12363.188605] Key type ._llcrypt registered [12363.192371] Key type .llcrypt registered [12363.328467] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [12363.737697] Key type .llcrypt unregistered [12363.739424] Key type ._llcrypt unregistered [12438.393924] Key type ._llcrypt registered [12438.398052] Key type .llcrypt registered [12438.569528] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [12438.982439] Key type .llcrypt unregistered [12438.985320] Key type ._llcrypt unregistered [12472.474693] Key type ._llcrypt registered [12472.479693] Key type .llcrypt registered [12472.690654] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [12473.090311] Key type .llcrypt unregistered [12473.091757] Key type ._llcrypt unregistered [12508.535805] Key type ._llcrypt registered [12508.542657] Key type .llcrypt registered [12508.670926] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [12508.950616] Key type .llcrypt unregistered [12508.954133] Key type ._llcrypt unregistered [12587.132643] Key type ._llcrypt registered [12587.134429] Key type .llcrypt registered [12587.230675] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [12587.487779] Key type .llcrypt unregistered [12587.489766] Key type ._llcrypt unregistered [12618.672347] Key type ._llcrypt registered [12618.674216] Key type .llcrypt registered [12618.783984] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [12619.072594] Key type .llcrypt unregistered [12619.074717] Key type ._llcrypt unregistered [12649.489541] Key type ._llcrypt registered [12649.491724] Key type .llcrypt registered [12649.577435] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [12649.839531] Key type .llcrypt unregistered [12649.847214] Key type ._llcrypt unregistered [12678.876473] Key type ._llcrypt registered [12678.883643] Key type .llcrypt registered [12678.987836] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [12679.260725] Key type .llcrypt unregistered [12679.262399] Key type ._llcrypt unregistered [12710.267721] Key type ._llcrypt registered [12710.270499] Key type .llcrypt registered [12710.388234] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [12710.701610] Key type .llcrypt unregistered [12710.706211] Key type ._llcrypt unregistered [12743.724925] Key type ._llcrypt registered [12743.730181] Key type .llcrypt registered [12743.875591] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [12744.241569] Key type .llcrypt unregistered [12744.246431] Key type ._llcrypt unregistered [12804.297512] Key type ._llcrypt registered [12804.303711] Key type .llcrypt registered [12804.408212] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [12804.600564] Key type .llcrypt unregistered [12804.603899] Key type ._llcrypt unregistered [12830.478987] Key type ._llcrypt registered [12830.487487] Key type .llcrypt registered [12830.592285] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [12830.887243] Key type .llcrypt unregistered [12830.889626] Key type ._llcrypt unregistered [12860.229770] Key type ._llcrypt registered [12860.230958] Key type .llcrypt registered [12860.313829] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [12860.513450] Key type .llcrypt unregistered [12860.514827] Key type ._llcrypt unregistered [12919.955993] Key type ._llcrypt registered [12919.958687] Key type .llcrypt registered [12920.073409] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [12938.029919] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [12940.088625] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12940.149044] alg: No test for adler32 (adler32-zlib) [12941.228412] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [12941.485484] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [12943.143235] Key type lgssc registered [12944.687401] Lustre: Echo OBD driver; http://www.lustre.org/ [12982.450780] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12982.488297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12983.792373] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [12983.811318] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [12983.907728] Lustre: tst250fs-MDT0000: new disk, initializing [12983.974490] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12983.992142] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12988.225613] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [12999.533857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12999.640451] Lustre: 373859:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12999.689184] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [12999.693168] Lustre: Skipped 1 previous similar message [12999.755329] Lustre: tst250fs-MDT0001: new disk, initializing [12999.892887] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12999.924057] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12999.937662] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13006.145410] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [13011.168698] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13017.885398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13018.248968] Lustre: tst250fs-OST0000: new disk, initializing [13018.253104] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13018.260982] Lustre: 375491:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13018.350164] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13021.963900] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13021.984490] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13022.121107] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13025.140222] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [13034.732651] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13034.866728] Lustre: tst250fs-OST0001: new disk, initializing [13034.869960] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13034.874689] Lustre: 376363:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13034.974283] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13040.735667] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13040.747734] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13040.854436] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13041.817545] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [13047.690253] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13050.332567] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [13050.336274] Lustre: Skipped 2 previous similar messages [13071.840982] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13071.849242] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13071.862912] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13071.930316] Lustre: Skipped 1 previous similar message [13071.951192] Lustre: Skipped 2 previous similar messages [13076.964214] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13076.978240] Lustre: Skipped 3 previous similar messages [13082.084667] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13082.097476] Lustre: Skipped 2 previous similar messages [13083.103118] Lustre: tst250fs-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13083.604578] Lustre: server umount tst250fs-MDT0000 complete [13087.694774] LustreError: 374662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785090973 with bad export cookie 1631614230644981024 [13087.711221] LustreError: 374662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13087.722806] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13088.239425] Lustre: server umount tst250fs-MDT0001 complete [13102.659420] Lustre: server umount tst250fs-OST0000 complete [13117.053918] Lustre: server umount tst250fs-OST0001 complete [13136.632216] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [13145.318079] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [13175.131676] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 14:37:39 (1785091059) [13177.450410] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13196.040442] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [13208.112544] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [13239.962536] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12950 sec ========= 14:38:44 (1785091124) [13241.876730] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 14:38:46 (1785091126) === [13245.626874] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 14:38:50 (1785091130) === [13291.981473] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [13294.745841] Key type lgssc unregistered [13295.069144] LNet: 385514:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13295.081727] LNetError: 385514:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13296.103658] LNet: Removed LNI 192.168.201.120@tcp [13297.111181] Key type .llcrypt unregistered [13297.113953] Key type ._llcrypt unregistered