[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 488240482 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001016] APIC: Switch to symmetric I/O mode setup [ 0.003388] x2apic enabled [ 0.004013] Switched APIC routing to physical x2apic. [ 0.005019] kvm-guest: setup PV IPIs [ 0.008380] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.010025] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011022] pid_max: default: 32768 minimum: 301 [ 0.012185] LSM: Security Framework initializing [ 0.013076] Yama: becoming mindful. [ 0.014062] SELinux: Initializing. [ 0.015094] *** VALIDATE selinux *** [ 0.023837] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027764] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029090] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031126] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032167] *** VALIDATE tmpfs *** [ 0.034441] *** VALIDATE proc *** [ 0.036195] *** VALIDATE cgroup *** [ 0.037022] *** VALIDATE cgroup2 *** [ 0.038329] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039194] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040016] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041042] Spectre V2 : User space: Vulnerable [ 0.042017] Speculative Store Bypass: Vulnerable [ 0.045594] debug: unmapping init [mem 0xffffffffb3259000-0xffffffffb3260fff] [ 0.047992] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048830] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049034] ... version: 2 [ 0.050019] ... bit width: 48 [ 0.051020] ... generic registers: 4 [ 0.052017] ... value mask: 0000ffffffffffff [ 0.053024] ... max period: 00007fffffffffff [ 0.054025] ... fixed-purpose events: 3 [ 0.055019] ... event mask: 000000070000000f [ 0.056408] rcu: Hierarchical SRCU implementation. [ 0.058842] smp: Bringing up secondary CPUs ... [ 0.059736] x86: Booting SMP configuration: [ 0.060038] .... node #0, CPUs: #1 #2 #3 [ 0.063662] smp: Brought up 1 node, 4 CPUs [ 0.065021] smpboot: Max logical packages: 1 [ 0.066026] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.251842] node 0 deferred pages initialised in 183ms [ 0.255150] devtmpfs: initialized [ 0.256359] x86/mm: Memory block size: 128MB [ 0.260000] gcov: version magic: 0x41383552 [ 0.262314] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.265137] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.267414] pinctrl core: initialized pinctrl subsystem [ 0.269218] [ 0.269773] ************************************************************* [ 0.272018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.274014] ** ** [ 0.276014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.278022] ** ** [ 0.280014] ** This means that this kernel is built to expose internal ** [ 0.282015] ** IOMMU data structures, which may compromise security on ** [ 0.285014] ** your system. ** [ 0.287014] ** ** [ 0.289013] ** If you see this message and you are not debugging the ** [ 0.291015] ** kernel, report this immediately to your vendor! ** [ 0.293012] ** ** [ 0.295016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.297013] ************************************************************* [ 0.299552] NET: Registered protocol family 16 [ 0.301479] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.304066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.307089] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.310067] cpuidle: using governor menu [ 0.311650] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.312501] PCI: Using configuration type 1 for base access [ 0.313000] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.322146] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.324040] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.326192] cryptd: max_cpu_qlen set to 1000 [ 0.329286] ACPI: Added _OSI(Module Device) [ 0.330019] ACPI: Added _OSI(Processor Device) [ 0.332014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.333011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.337083] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.342350] ACPI: Interpreter enabled [ 0.343000] ACPI: PM: (supports S0 S3 S4 S5) [ 0.345013] ACPI: Using IOAPIC for interrupt routing [ 0.346114] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.348405] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.357335] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.358034] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.360015] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.362084] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.366422] acpiphp: Slot [2] registered [ 0.367106] acpiphp: Slot [5] registered [ 0.369101] acpiphp: Slot [6] registered [ 0.370124] acpiphp: Slot [7] registered [ 0.371118] acpiphp: Slot [8] registered [ 0.372128] acpiphp: Slot [9] registered [ 0.373104] acpiphp: Slot [10] registered [ 0.375145] acpiphp: Slot [3] registered [ 0.376091] acpiphp: Slot [4] registered [ 0.377100] acpiphp: Slot [11] registered [ 0.378121] acpiphp: Slot [12] registered [ 0.379087] acpiphp: Slot [13] registered [ 0.381120] acpiphp: Slot [14] registered [ 0.382090] acpiphp: Slot [15] registered [ 0.383086] acpiphp: Slot [16] registered [ 0.384100] acpiphp: Slot [17] registered [ 0.386124] acpiphp: Slot [18] registered [ 0.387096] acpiphp: Slot [19] registered [ 0.388087] acpiphp: Slot [20] registered [ 0.389096] acpiphp: Slot [21] registered [ 0.390086] acpiphp: Slot [22] registered [ 0.392087] acpiphp: Slot [23] registered [ 0.393109] acpiphp: Slot [24] registered [ 0.395160] acpiphp: Slot [25] registered [ 0.396121] acpiphp: Slot [26] registered [ 0.398129] acpiphp: Slot [27] registered [ 0.399108] acpiphp: Slot [28] registered [ 0.400106] acpiphp: Slot [29] registered [ 0.402120] acpiphp: Slot [30] registered [ 0.403124] acpiphp: Slot [31] registered [ 0.405103] PCI host bridge to bus 0000:00 [ 0.406020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.409033] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.411038] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.414035] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.417029] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.419028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.423174] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.426988] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.430346] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.439800] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.443333] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.445015] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.446012] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.448012] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.449571] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.452872] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.455060] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.457735] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.461745] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.470032] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.475025] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.480000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.486019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.494023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.504027] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.512279] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.518017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.523021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.542019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.552013] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.561018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.567025] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.589030] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.600203] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.612022] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.622022] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.640022] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.652142] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.662021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.672016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.689021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.702171] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.710019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.734031] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.752020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.765066] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.766410] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.768361] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.771390] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.773167] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.777117] iommu: Default domain type: Passthrough [ 0.778482] SCSI subsystem initialized [ 0.779140] ACPI: bus type USB registered [ 0.781138] usbcore: registered new interface driver usbfs [ 0.782084] usbcore: registered new interface driver hub [ 0.784108] usbcore: registered new device driver usb [ 0.786185] pps_core: LinuxPPS API ver. 1 registered [ 0.788013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.790082] PTP clock support registered [ 0.792145] EDAC MC: Ver: 3.0.0 [ 0.794132] PCI: Using ACPI for IRQ routing [ 0.796906] NetLabel: Initializing [ 0.798023] NetLabel: domain hash size = 128 [ 0.800012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.802267] NetLabel: unlabeled traffic allowed by default [ 0.804152] vgaarb: loaded [ 0.806317] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.807014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.811283] clocksource: Switched to clocksource kvm-clock [ 0.921689] VFS: Disk quotas dquot_6.6.0 [ 0.923440] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.926188] *** VALIDATE ramfs *** [ 0.927526] *** VALIDATE hugetlbfs *** [ 0.929035] pnp: PnP ACPI init [ 0.931411] pnp: PnP ACPI: found 6 devices [ 0.949177] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.952504] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.954809] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.957279] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.960021] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.962550] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.965626] NET: Registered protocol family 2 [ 0.970406] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.975360] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.978319] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.983119] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.986839] TCP: Hash tables configured (established 65536 bind 65536) [ 0.988852] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.990829] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.993353] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.995349] NET: Registered protocol family 1 [ 0.997816] RPC: Registered named UNIX socket transport module. [ 0.999338] RPC: Registered udp transport module. [ 1.000655] RPC: Registered tcp transport module. [ 1.002167] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.003917] NET: Registered protocol family 44 [ 1.005025] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.006303] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.007632] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.009296] PCI: CLS 0 bytes, default 64 [ 1.010753] Unpacking initramfs... [ 2.416209] debug: unmapping init [mem 0xffff891b7cc54000-0xffff891b7ffbffff] [ 2.423221] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.426449] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.430162] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.899119] Initialise system trusted keyrings [ 2.902246] Key type blacklist registered [ 2.904228] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.915449] zbud: loaded [ 2.919335] *** VALIDATE nfs *** [ 2.920851] *** VALIDATE nfs4 *** [ 2.922305] pstore: using deflate compression [ 2.925923] Platform Keyring initialized [ 3.019680] NET: Registered protocol family 38 [ 3.020957] Key type asymmetric registered [ 3.022178] Asymmetric key parser 'x509' registered [ 3.023298] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.025312] io scheduler mq-deadline registered [ 3.027149] io scheduler kyber registered [ 3.028744] io scheduler bfq registered [ 3.031251] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.033883] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.037171] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.040290] ACPI: Power Button [PWRF] [ 3.045269] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.052591] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.064553] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.071663] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.088324] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.115383] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.142248] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.148109] Non-volatile memory driver v1.3 [ 3.150032] Linux agpgart interface v0.103 [ 3.180330] virtio_blk virtio1: [vda] 149944 512-byte logical blocks (76.8 MB/73.2 MiB) [ 3.184166] vda: detected capacity change from 0 to 76771328 [ 3.198747] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.201935] vdb: detected capacity change from 0 to 1073741824 [ 3.219774] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.223357] vdc: detected capacity change from 0 to 2621440000 [ 3.239151] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.241375] vdd: detected capacity change from 0 to 2621440000 [ 3.253363] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.255829] vde: detected capacity change from 0 to 4294967296 [ 3.269615] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.271882] vdf: detected capacity change from 0 to 4294967296 [ 3.280726] libphy: Fixed MDIO Bus: probed [ 3.286766] usbcore: registered new interface driver usbserial_generic [ 3.288581] usbserial: USB Serial support registered for generic [ 3.290040] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.293712] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.294945] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.297224] mousedev: PS/2 mouse device common for all mice [ 3.300084] rtc_cmos 00:05: RTC can wake from S4 [ 3.302121] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.305580] rtc_cmos 00:05: registered as rtc0 [ 3.306550] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.307482] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.312986] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.313639] intel_pstate: CPU model not supported [ 3.319431] hid: raw HID events driver (C) Jiri Kosina [ 3.321325] usbcore: registered new interface driver usbhid [ 3.323049] usbhid: USB HID core driver [ 3.324195] drop_monitor: Initializing network drop monitor service [ 3.326108] Initializing XFRM netlink socket [ 3.328097] NET: Registered protocol family 10 [ 3.330626] Segment Routing with IPv6 [ 3.332076] NET: Registered protocol family 17 [ 3.334199] mpls_gso: MPLS GSO support [ 3.339391] RAS: Correctable Errors collector initialized. [ 3.341377] AVX version of gcm_enc/dec engaged. [ 3.342790] AES CTR mode by8 optimization enabled [ 3.407849] sched_clock: Marking stable (3407826516, 0)->(4406601501, -998774985) [ 3.411091] registered taskstats version 1 [ 3.413126] Loading compiled-in X.509 certificates [ 3.414574] zswap: loaded using pool lzo/zbud [ 3.433422] Key type big_key registered [ 3.442672] Key type encrypted registered [ 3.443800] ima: No TPM chip found, activating TPM-bypass! [ 3.445120] ima: Allocated hash algorithm: sha1 [ 3.446207] ima: No architecture policies found [ 3.447300] evm: Initialising EVM extended attributes: [ 3.448646] evm: security.selinux [ 3.450087] evm: security.ima [ 3.451266] evm: security.capability [ 3.452727] evm: HMAC attrs: 0x1 [ 3.455148] rtc_cmos 00:05: setting system clock to 2026-09-07 16:11:13 UTC (1788797473) [ 3.461355] debug: unmapping init [mem 0xffffffffb4203000-0xffffffffb43fffff] [ 3.464645] debug: unmapping init [mem 0xffffffffb2f82000-0xffffffffb3258fff] [ 3.472075] Write protecting the kernel read-only data: 28672k [ 3.474847] debug: unmapping init [mem 0xffffffffb1603000-0xffffffffb17fffff] [ 3.476664] debug: unmapping init [mem 0xffffffffb1f14000-0xffffffffb1ffffff] [ 3.507759] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.516417] systemd[1]: Detected virtualization kvm. [ 3.518740] systemd[1]: Detected architecture x86-64. [ 3.520920] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.542170] systemd[1]: No hostname configured. [ 3.544374] systemd[1]: Set hostname to . [ 3.546351] random: systemd: uninitialized urandom read (16 bytes read) [ 3.548866] systemd[1]: Initializing machine ID from random generator. [ 3.588815] random: ln: uninitialized urandom read (6 bytes read) [ 3.651423] random: systemd: uninitialized urandom read (16 bytes read) [ 3.652980] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.656298] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.659098] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... Starting Setup Virtual Console... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.295493] device-mapper: uevent: version 1.0.3 [ 4.297385] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.947163] virtio_net virtio0 ens2: renamed from eth0 [ 4.967798] random: fast init done [ 5.037766] scsi host0: ata_piix [ 5.042634] scsi host1: ata_piix [ 5.044662] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.047653] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.350808] dracut-initqueue[588]: RTNETLINK answers: File exists [ 9.965189] random: crng init done [ 9.967136] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.214439] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.310286] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.547731] SELinux: Disabled at runtime. [ 11.607442] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.614172] systemd[1]: Detected virtualization kvm. [ 11.615902] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.096360] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.101892] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.110154] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.114970] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.119242] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.133133] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.139647] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ 12.194866] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Remount Root and Kernel File Systems... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Control Socket. Mounting POSIX Message Queue File System... [ OK ] Reached target rpc_pipefs.target. Starting Apply Kernel Variables... Mounting Huge Pages File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.533761] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.826453] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.831655] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.967472] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.979544] EDAC sbridge: Ver: 1.1.2 [ 14.271239] Key type dns_resolver registered [ 14.561919] NFS: Registering the id_resolver key type [ 14.563970] Key type id_resolver registered [ 14.565734] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg152-server login: [ 36.411340] hrtimer: interrupt took 1747157 ns [ 82.796952] libcfs: loading out-of-tree module taints kernel. [ 82.865528] Key type ._llcrypt registered [ 82.867116] Key type .llcrypt registered [ 82.966511] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 101.829704] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 103.763913] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 103.791343] alg: No test for adler32 (adler32-zlib) [ 105.278915] Lustre: Lustre: Build Version: 2.17.58_39_gc924a9e [ 106.239796] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 108.127362] Key type lgssc registered [ 109.903885] Lustre: Echo OBD driver; http://www.lustre.org/ [ 128.740093] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 173.717677] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 187.674873] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 187.721542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 189.143445] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 189.180919] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 189.293644] Lustre: lustre-MDT0000: new disk, initializing [ 189.407608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 189.437777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 193.534975] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 210.353168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 210.537565] Lustre: 6518:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 210.584514] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 210.592303] Lustre: Skipped 1 previous similar message [ 210.740542] Lustre: lustre-MDT0001: new disk, initializing [ 210.858227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 210.940274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 210.951555] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 216.354922] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 221.545988] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 231.051732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 231.453621] Lustre: lustre-OST0000: new disk, initializing [ 231.456984] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 231.462030] Lustre: 8456:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 231.536496] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 235.647207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 235.663883] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 235.758140] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 238.779229] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 255.000318] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 255.244319] Lustre: lustre-OST0001: new disk, initializing [ 255.256822] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 255.266677] Lustre: 9529:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 255.391224] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 260.672992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 260.681840] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 260.762369] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 262.307354] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 275.672130] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 283.265930] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 291.379965] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing check_logdir /tmp/testlogs/ [ 297.030496] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing yml_node [ 302.444836] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 305.676577] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 308.284209] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 310.169553] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Sep 7 12:16:19 EDT 2026 [ 327.286618] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 328.904774] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 347.617580] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 347.620590] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 347.637068] Lustre: Skipped 3 previous similar messages [ 347.638168] Lustre: Skipped 2 previous similar messages [ 352.738996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 352.754100] Lustre: Skipped 4 previous similar messages [ 357.857563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 357.865967] Lustre: Skipped 3 previous similar messages [ 359.391153] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 359.966769] Lustre: server umount lustre-MDT0000 complete [ 364.209685] LustreError: 6511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788797834 with bad export cookie 13508063444168294770 [ 364.230523] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 368.123156] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 368.123552] LustreError: 6525:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 368.123854] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 368.149852] Lustre: Skipped 1 previous similar message [ 368.176660] LustreError: 6525:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 369.133034] LustreError: 6526:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 369.148660] LustreError: 6526:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 370.962342] Lustre: server umount lustre-MDT0001 complete [ 375.789804] Lustre: server umount lustre-OST0000 complete [ 380.245862] Lustre: server umount lustre-OST0001 complete [ 387.071191] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 397.778584] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 435.044566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 435.496344] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 435.536302] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 435.669296] Lustre: lustre-MDT0000: new disk, initializing [ 435.763380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 435.787388] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 440.878284] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 454.472505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 454.633968] Lustre: 15085:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 454.701539] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 454.706431] Lustre: Skipped 1 previous similar message [ 454.824338] Lustre: lustre-MDT0001: new disk, initializing [ 454.950234] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 454.999096] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 455.022705] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 458.854842] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 469.344208] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 476.044521] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 483.993119] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 484.398610] Lustre: lustre-OST0000: new disk, initializing [ 484.407878] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 484.413953] Lustre: 17053:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 484.537333] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 485.857507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 485.866090] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 485.992690] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 490.967796] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 500.993958] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 506.143760] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 506.353459] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 510.692720] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 510.924832] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 516.585502] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 516.587503] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 516.612395] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 516.621329] Lustre: Skipped 4 previous similar messages [ 518.826153] Lustre: server umount lustre-OST0000 complete [ 526.820942] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 526.824584] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 526.843201] Lustre: Skipped 1 previous similar message [ 526.852196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 526.863763] Lustre: Skipped 1 previous similar message [ 528.938961] Lustre: server umount lustre-MDT0000 complete [ 531.939122] LustreError: 15096:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 531.964642] LustreError: 15096:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 532.617295] LustreError: 15075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788798002 with bad export cookie 13508063444168296303 [ 532.625373] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 532.635190] LustreError: 15075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 533.112822] Lustre: server umount lustre-MDT0001 complete [ 539.613542] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 12:20:08 (1788798008) [ 559.877038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 560.347393] LustreError: 19928:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 560.436522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 565.448845] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 565.731567] LustreError: 19929:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 574.731573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 579.164463] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 587.642993] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 594.111875] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 596.184320] Lustre: Failing over lustre-MDT0000 [ 596.505301] Lustre: server umount lustre-MDT0000 complete [ 600.289294] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 600.291381] LustreError: 19929:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 600.308364] Lustre: Skipped 1 previous similar message [ 600.352553] LustreError: 19929:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 616.009216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 616.089552] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 616.319828] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 620.387856] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 620.394560] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 620.408739] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 620.823102] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 630.757177] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 634.820070] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 634.824159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 634.837496] Lustre: Skipped 1 previous similar message [ 634.850101] Lustre: Skipped 1 previous similar message [ 649.184370] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 649.408113] Lustre: server umount lustre-MDT0000 complete [ 650.209698] LustreError: 19929:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 650.221364] LustreError: 19929:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 652.787444] LustreError: 19909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788798122 with bad export cookie 13508063444168297717 [ 652.793935] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 653.075566] Lustre: server umount lustre-MDT0001 complete [ 662.321886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 663.132291] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 663.145215] Lustre: Skipped 2 previous similar messages [ 669.114554] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 679.285893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 685.172875] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 693.674816] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 701.793649] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 705.064200] Lustre: Failing over lustre-MDT0000 [ 705.505144] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 705.512513] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 705.534293] LustreError: 23265:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 705.553659] LustreError: 23265:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 705.627407] Lustre: server umount lustre-MDT0000 complete [ 724.965114] Lustre: 3652:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788798179/real 1788798179] req@ffff891ac3780e00 x1875690207250304/t0(0) o400->MGC192.168.201.152@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788798195 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 725.000218] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 728.242567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 735.226361] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xbb7648223917509f [ 735.249592] Lustre: MGC192.168.201.152@tcp: Connection restored to 0@lo (at 0@lo) [ 735.260073] Lustre: Skipped 1 previous similar message [ 735.749467] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 740.837118] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 740.864282] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 740.892409] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 741.690434] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 752.349676] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 756.357614] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 756.373590] Lustre: Skipped 1 previous similar message [ 756.394970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 756.408017] Lustre: Skipped 4 previous similar messages [ 761.312948] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 763.056466] Lustre: server umount lustre-MDT0000 complete [ 767.900834] LustreError: 23246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788798237 with bad export cookie 13508063444168298655 [ 767.908266] LustreError: 23246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 767.916930] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 768.285889] Lustre: server umount lustre-MDT0001 complete [ 777.528218] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 12:24:06 (1788798246) [ 795.008833] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 810.481201] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 852.917753] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 866.000333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 866.217209] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 866.242609] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 866.351694] Lustre: lustre-MDT0000: new disk, initializing [ 866.446934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 866.458873] Lustre: Skipped 2 previous similar messages [ 866.483552] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 871.099495] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 882.090596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 882.215841] Lustre: 29666:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 882.425868] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 887.052427] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 897.432434] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 903.250885] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 909.335722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 909.601733] Lustre: 31637:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 910.871412] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 910.937687] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 910.998153] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 911.016899] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 915.844719] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 925.950805] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 931.470824] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 951.782566] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 951.792926] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 951.806788] Lustre: Skipped 1 previous similar message [ 951.815922] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 951.818898] Lustre: Skipped 2 previous similar messages [ 954.578926] Lustre: server umount lustre-OST0000 complete [ 964.920691] Lustre: server umount lustre-MDT0000 complete [ 967.654237] LustreError: 32024:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 967.668662] LustreError: 32024:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 17 previous similar messages [ 969.047155] LustreError: 30460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788798439 with bad export cookie 13508063444168299068 [ 969.052328] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 969.059139] LustreError: 30460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 969.604989] Lustre: server umount lustre-MDT0001 complete [ 981.382897] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 984.833558] Key type lgssc unregistered [ 985.086623] LNet: 33614:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 985.094829] LNetError: 33614:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 985.115720] LNet: Removed LNI 192.168.201.152@tcp [ 986.029368] Key type .llcrypt unregistered [ 986.037671] Key type ._llcrypt unregistered [ 1010.628798] Key type ._llcrypt registered [ 1010.631235] Key type .llcrypt registered [ 1010.767241] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 1012.271177] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1012.388156] alg: No test for adler32 (adler32-zlib) [ 1013.413840] Lustre: Lustre: Build Version: 2.17.58_39_gc924a9e [ 1013.608861] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 1015.288330] Key type lgssc registered [ 1016.474849] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1024.242251] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1024.258239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1025.658721] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1029.924612] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1035.234991] LustreError: 35024:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1035.264738] LustreError: 35024:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1037.310607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1037.632760] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1041.846720] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1048.586594] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1054.188712] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1061.240140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1061.642126] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1062.717527] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1067.747483] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1070.977034] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1075.372590] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1091.009220] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 12:29:20 (1788798560) [ 1093.307312] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1095.594127] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1104.682294] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 12:29:33 (1788798573) [ 1114.222912] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 12:29:43 (1788798583) [ 1115.197906] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1124.470460] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1124.477034] Lustre: Skipped 2 previous similar messages [ 1126.414706] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 12:29:55 (1788798595) [ 1130.523328] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1130.529769] Lustre: Skipped 3 previous similar messages [ 1150.515827] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1150.521149] Lustre: Skipped 6 previous similar messages [ 1154.122709] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 12:30:22 (1788798622) [ 1157.428647] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1161.814116] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1170.751443] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 12:30:39 (1788798639) [ 1172.916186] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1172.919119] Lustre: Skipped 3 previous similar messages [ 1186.338213] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 12:30:55 (1788798655) [ 1201.123224] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1201.135699] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1201.160209] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1204.196516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1204.201759] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1204.209700] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1206.914840] Lustre: server umount lustre-OST0000 complete [ 1211.873192] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1211.883107] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1211.895578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1211.902173] Lustre: Skipped 1 previous similar message [ 1214.434616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1214.447650] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1216.606702] Lustre: server umount lustre-MDT0000 complete [ 1219.553160] LustreError: 35020:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1219.562798] LustreError: 35020:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1220.928795] LustreError: 35004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788798690 with bad export cookie 5573713967957543943 [ 1220.944856] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1221.792877] Lustre: server umount lustre-MDT0001 complete [ 1235.266159] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 1238.301991] Key type lgssc unregistered [ 1238.649677] LNet: 42878:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1238.659059] LNetError: 42878:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1238.686655] LNet: Removed LNI 192.168.201.152@tcp [ 1239.540786] Key type .llcrypt unregistered [ 1239.544092] Key type ._llcrypt unregistered [ 1259.151847] Key type ._llcrypt registered [ 1259.155103] Key type .llcrypt registered [ 1259.282474] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 1275.762424] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 1277.006932] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1277.017870] alg: No test for adler32 (adler32-zlib) [ 1278.205892] Lustre: Lustre: Build Version: 2.17.58_39_gc924a9e [ 1278.462076] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 1280.135244] Key type lgssc registered [ 1281.370059] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1315.621077] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1315.662792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1317.078913] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1317.155711] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1317.372403] Lustre: lustre-MDT0000: new disk, initializing [ 1317.568443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1317.600994] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1323.275426] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1335.792196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1335.938140] Lustre: 46027:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1335.989488] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1335.992487] Lustre: Skipped 1 previous similar message [ 1336.047807] Lustre: lustre-MDT0001: new disk, initializing [ 1336.120412] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1336.152189] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1336.169287] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1341.440678] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1352.653634] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1358.934869] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1367.106793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1367.466271] Lustre: lustre-OST0000: new disk, initializing [ 1367.470341] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1367.476992] Lustre: 48001:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1367.616691] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1373.312791] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1373.327691] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1373.509143] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1375.555169] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1386.247119] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1390.627726] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1390.822829] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1396.344926] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1396.757196] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1404.396250] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1404.397382] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1404.399816] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1404.410688] Lustre: Skipped 1 previous similar message [ 1404.420730] LustreError: Skipped 1 previous similar message [ 1405.754495] Lustre: server umount lustre-OST0000 complete [ 1413.094026] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1413.106340] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1413.119152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1413.122021] Lustre: Skipped 1 previous similar message [ 1414.627098] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1414.627345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1414.651404] Lustre: Skipped 1 previous similar message [ 1416.007148] Lustre: server umount lustre-MDT0000 complete [ 1420.159330] LustreError: 46020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788798890 with bad export cookie 6139213328892013539 [ 1420.168082] LustreError: 46020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1420.173977] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1420.657977] Lustre: server umount lustre-MDT0001 complete [ 1432.939432] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 12:35:01 (1788798901) [ 1440.628395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1441.106568] LustreError: 50462:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1441.136149] LustreError: 50462:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1441.258823] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1446.373914] LustreError: 50463:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1448.084401] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1451.489168] LustreError: 50462:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1456.609870] LustreError: 50463:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1457.234532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1457.669812] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1462.109711] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1470.435189] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1476.861195] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1484.284389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1484.623649] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1490.829986] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1498.229616] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1505.668845] LustreError: 52990:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1506.583735] Lustre: Modifying parameter general.timeout=1 in log params [ 1508.392213] Lustre: Modifying parameter general.timeout=3 in log params [ 1508.399661] Lustre: Skipped 1 previous similar message [ 1511.070486] Lustre: Modifying parameter general.timeout=6 in log params [ 1511.075825] Lustre: Skipped 2 previous similar messages [ 1515.544143] Lustre: Modifying parameter general.timeout=11 in log params [ 1515.552122] Lustre: Skipped 4 previous similar messages [ 1523.616900] Lustre: Modifying parameter general.timeout=19 in log params [ 1523.624322] Lustre: Skipped 7 previous similar messages [ 1540.140976] Lustre: Modifying parameter general.timeout=36 in log params [ 1540.142739] Lustre: Skipped 16 previous similar messages [ 1572.500835] Lustre: Modifying parameter general.timeout=66 in log params [ 1572.505600] Lustre: Skipped 29 previous similar messages [ 1637.065657] Lustre: Modifying parameter general.timeout=121 in log params [ 1637.071992] Lustre: Skipped 54 previous similar messages [ 1659.992409] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 12:38:48 (1788799128) [ 1663.086353] Lustre: 59953:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1665.066656] Lustre: 60052:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1669.332179] Lustre: 60265:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1669.348073] Lustre: 60265:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1674.086843] Lustre: 60461:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1674.107890] Lustre: 60461:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1710.338453] Lustre: 61838:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1710.376156] LustreError: 61838:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1716.249033] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 12:39:44 (1788799184) [ 1734.120346] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1734.121831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1734.131799] Lustre: Skipped 1 previous similar message [ 1734.141126] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1736.566725] Lustre: server umount lustre-MDT0000 complete [ 1739.234931] LustreError: 51190:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1739.272512] LustreError: 51190:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1740.340525] LustreError: 50442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788799210 with bad export cookie 6139213328892014421 [ 1740.356622] LustreError: 50442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1740.356522] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1740.685796] Lustre: server umount lustre-MDT0001 complete [ 1755.387251] Lustre: server umount lustre-OST0000 complete [ 1784.400039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1784.720824] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1784.738506] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1784.744931] Lustre: Skipped 31 previous similar messages [ 1785.046864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1790.019986] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1800.123450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1800.203955] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1800.232811] Lustre: 64371:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1807.028600] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1816.835825] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1824.158040] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1832.469044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1832.627949] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1832.984502] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1832.988753] Lustre: Skipped 1 previous similar message [ 1840.518584] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1849.985071] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1851.639781] Lustre: 66937:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1864.160850] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1864.168407] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1864.176887] Lustre: Skipped 1 previous similar message [ 1864.188418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1864.191179] Lustre: Skipped 2 previous similar messages [ 1865.653854] Lustre: server umount lustre-OST0000 complete [ 1874.403654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1874.413681] Lustre: Skipped 4 previous similar messages [ 1875.581735] Lustre: server umount lustre-MDT0000 complete [ 1879.525794] LustreError: 44249:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000f4e579cc x1875691436225024/t0(0) o38->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1879.531326] LustreError: 64362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788799349 with bad export cookie 6139213328892017662 [ 1879.553339] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1879.563344] LustreError: 64362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 1879.796135] Lustre: server umount lustre-MDT0001 complete [ 1889.943334] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 1892.419597] Key type lgssc unregistered [ 1892.735843] LNet: 68147:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1892.753558] LNetError: 68147:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1893.801787] LNet: Removed LNI 192.168.201.152@tcp [ 1895.001228] Key type .llcrypt unregistered [ 1895.009366] Key type ._llcrypt unregistered [ 1927.003675] Key type ._llcrypt registered [ 1927.005882] Key type .llcrypt registered [ 1927.087486] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 1928.591630] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1928.642438] alg: No test for adler32 (adler32-zlib) [ 1929.915834] Lustre: Lustre: Build Version: 2.17.58_39_gc924a9e [ 1930.539893] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 1932.263231] Key type lgssc registered [ 1934.033147] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1944.759279] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1944.797556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1946.937651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1953.027985] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1956.333252] LustreError: 69561:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1956.370211] LustreError: 69561:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1961.444756] LustreError: 69562:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1962.503065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1963.226532] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1967.348819] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1975.142164] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1981.292355] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1989.009397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1989.625784] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1996.302664] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2004.066405] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2015.139625] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2024.418363] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2024.429162] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2024.448350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2027.488133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2027.490931] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2027.528786] Lustre: Skipped 1 previous similar message [ 2029.019114] Lustre: server umount lustre-MDT0000 complete [ 2032.620403] LustreError: 69557:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2032.630175] LustreError: 69557:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2032.730537] LustreError: 69543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788799502 with bad export cookie 17326388394421387976 [ 2032.730988] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2032.748705] LustreError: 69543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2033.114896] Lustre: server umount lustre-MDT0001 complete [ 2047.529633] Lustre: server umount lustre-OST0000 complete [ 2077.611369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2078.087706] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2078.133241] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2078.462366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2083.632622] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2092.152450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2092.247844] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2092.277836] Lustre: 74433:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2092.856901] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2097.940944] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2106.026558] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2113.161823] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2121.054972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2121.303469] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2121.776389] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2128.257970] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2135.868474] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2137.232257] Lustre: 77008:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2139.062501] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2147.808520] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2147.812794] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2147.844485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2147.857859] Lustre: Skipped 2 previous similar messages [ 2150.806917] Lustre: server umount lustre-OST0000 complete [ 2158.565113] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2158.573940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2158.580873] Lustre: Skipped 2 previous similar messages [ 2158.596691] Lustre: Skipped 3 previous similar messages [ 2162.364662] Lustre: server umount lustre-MDT0000 complete [ 2163.697837] LustreError: 74445:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2163.741760] LustreError: 74445:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2167.355936] LustreError: 74425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788799637 with bad export cookie 17326388394421388809 [ 2167.361256] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2167.377264] LustreError: 74425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2168.235710] Lustre: server umount lustre-MDT0001 complete [ 2185.267504] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 2188.800958] Key type lgssc unregistered [ 2189.165846] LNet: 78222:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2189.170397] LNetError: 78222:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2189.185646] LNet: Removed LNI 192.168.201.152@tcp [ 2189.960227] Key type .llcrypt unregistered [ 2189.967362] Key type ._llcrypt unregistered [ 2215.447690] Key type ._llcrypt registered [ 2215.451348] Key type .llcrypt registered [ 2215.582157] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 2216.793293] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2217.004136] alg: No test for adler32 (adler32-zlib) [ 2218.125864] Lustre: Lustre: Build Version: 2.17.58_39_gc924a9e [ 2218.418664] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 2220.127440] Key type lgssc registered [ 2221.628206] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2229.090278] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2229.120095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2230.571078] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2236.010976] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2239.995638] LustreError: 79631:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2240.018925] LustreError: 79631:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2244.639737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2245.411443] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2251.222428] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2260.281988] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2267.434713] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2275.120457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2275.296888] LustreError: 81461:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2275.683928] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2282.286251] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2289.675991] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2296.349266] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2297.174333] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2299.029149] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2299.032736] Lustre: Skipped 1 previous similar message [ 2301.657550] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2301.661431] Lustre: Skipped 2 previous similar messages [ 2305.875842] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2305.879850] Lustre: Skipped 4 previous similar messages [ 2314.733575] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2314.737411] Lustre: Skipped 8 previous similar messages [ 2331.332279] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2331.339791] Lustre: Skipped 18 previous similar messages [ 2363.789245] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2363.798046] Lustre: Skipped 26 previous similar messages [ 2399.950840] Lustre: 87737:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2402.792910] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 12:51:12 (1788799872) [ 2404.260944] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2406.734432] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 12:51:15 (1788799875) [ 2413.035261] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2413.040290] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2413.042377] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2413.048150] Lustre: Skipped 1 previous similar message [ 2416.921637] Lustre: server umount lustre-OST0000 complete [ 2423.269886] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2423.273456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2423.294456] Lustre: Skipped 1 previous similar message [ 2423.307958] Lustre: Skipped 2 previous similar messages [ 2426.892456] Lustre: server umount lustre-MDT0000 complete [ 2428.387386] LustreError: 79625:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2428.422629] LustreError: 79625:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2429.862925] LustreError: 79611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788799899 with bad export cookie 8450548134515338727 [ 2429.879408] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2429.883700] LustreError: 79611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2430.284462] Lustre: server umount lustre-MDT0001 complete [ 2440.895553] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 2443.877151] Key type lgssc unregistered [ 2444.191676] LNet: 88985:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2444.210468] LNetError: 88985:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2445.221778] LNet: Removed LNI 192.168.201.152@tcp [ 2446.515971] Key type .llcrypt unregistered [ 2446.519537] Key type ._llcrypt unregistered [ 2452.813299] Key type ._llcrypt registered [ 2452.817186] Key type .llcrypt registered [ 2452.989136] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2459.238329] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules [ 2461.055514] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2461.135760] alg: No test for adler32 (adler32-zlib) [ 2462.340575] Lustre: Lustre: Build Version: 2.17.58_39_gc924a9e [ 2462.535285] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 2464.231267] Key type lgssc registered [ 2465.737511] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2466.964611] LustreError: 90232:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2486.997603] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 2489.881495] LustreError: 90232:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2496.645544] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2496.659117] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2498.155356] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2502.681983] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2508.773113] LustreError: 91260:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2508.792514] LustreError: 91260:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2511.627915] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 12:53:00 (1788799980) [ 2513.898664] LustreError: 91261:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2516.549023] LustreError: 91292:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2516.560941] LustreError: 91292:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2516.568808] LustreError: 91292:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 2516.888963] Lustre: server umount lustre-MDT0000 complete [ 2529.159605] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 2531.784939] Key type lgssc unregistered [ 2532.070859] LNet: 92785:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2532.083251] LNetError: 92785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2532.113826] LNet: Removed LNI 192.168.201.152@tcp [ 2532.876124] Key type .llcrypt unregistered [ 2532.879293] Key type ._llcrypt unregistered [ 2549.771452] Key type ._llcrypt registered [ 2549.773475] Key type .llcrypt registered [ 2549.870747] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 2551.403557] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2551.461461] alg: No test for adler32 (adler32-zlib) [ 2552.518552] Lustre: Lustre: Build Version: 2.17.58_39_gc924a9e [ 2552.754248] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 2554.456508] Key type lgssc registered [ 2555.441404] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2562.467317] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2562.507539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2564.135127] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2568.660753] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2574.310839] LustreError: 93999:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2574.323245] LustreError: 93999:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2577.023738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2577.645773] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2582.213434] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2590.030715] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2595.678100] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2602.280796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2602.480568] LustreError: 95829:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2603.304833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2610.629488] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2619.291208] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2662.407356] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 12:55:31 (1788800131) [ 2664.421445] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2666.367801] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 12:55:35 (1788800135) [ 2671.073920] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2671.077479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2671.091401] Lustre: Skipped 1 previous similar message [ 2671.102796] Lustre: Skipped 2 previous similar messages [ 2676.195442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2676.202377] Lustre: Skipped 2 previous similar messages [ 2677.091783] Lustre: server umount lustre-MDT0000 complete [ 2680.982768] LustreError: 93979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788800151 with bad export cookie 16646325704053704981 [ 2680.984862] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2680.997227] LustreError: 93979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2681.318300] LustreError: 93995:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2681.335706] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2681.349688] Lustre: Skipped 1 previous similar message [ 2681.446392] Lustre: server umount lustre-MDT0001 complete [ 2687.681851] Lustre: server umount lustre-OST0000 complete [ 2698.791542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2699.136164] LustreError: 98275:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2699.158350] LustreError: 98275:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2699.329906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2703.952835] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2704.356961] LustreError: 98276:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2713.247454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2713.757540] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2717.399908] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2725.080315] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2730.677372] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2744.684161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2744.898678] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2744.906372] LustreError: 98264:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2744.930776] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2744.938989] LustreError: 100358:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2744.944408] LustreError: 100358:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2744.951380] LustreError: 100358:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2745.034863] Lustre: server umount lustre-OST0000 complete [ 2745.039686] LustreError: 100358:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2753.102379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2753.231537] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2753.239961] LustreError: 98264:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2753.260843] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2753.269710] LustreError: 100825:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2753.275877] LustreError: 100825:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2753.282780] LustreError: 100825:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2753.362326] Lustre: server umount lustre-OST0000 complete [ 2753.371892] LustreError: 100825:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2762.360714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2762.590716] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2762.599327] Lustre: Found index 0 for lustre-OST0000, updating log [ 2762.617553] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2762.746546] Lustre: lustre-OST0000: new disk, initializing [ 2762.753428] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2762.769898] Lustre: 101360:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2763.147720] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2770.195273] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2774.304814] Lustre: Failing over lustre-OST0000 [ 2774.499851] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2774.503534] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2774.527109] LustreError: 101661:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2774.538881] LustreError: 101661:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2774.683490] Lustre: server umount lustre-OST0000 complete [ 2779.617273] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2779.619502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2779.635030] Lustre: Skipped 1 previous similar message [ 2779.648836] Lustre: Skipped 1 previous similar message [ 2783.722235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2783.739422] Lustre: Skipped 1 previous similar message [ 2784.833555] Lustre: server umount lustre-MDT0000 complete [ 2788.573076] LustreError: 98256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788800258 with bad export cookie 16646325704053707725 [ 2788.576678] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2788.599500] LustreError: 98256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2789.035633] Lustre: server umount lustre-MDT0001 complete [ 2798.797146] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 12:57:47 (1788800267) [ 2816.373819] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 2826.732286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2827.287494] LustreError: 103706:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2827.510151] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2832.549647] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2841.391058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2846.783608] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2850.613782] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2857.835205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2858.558588] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2864.787325] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2866.722746] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2866.725813] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2866.748077] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2871.782227] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 2871.805302] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2872.820236] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:03 [ 2872.855663] Lustre: lustre-OST0000: Recovery over after 0:06, of 1 clients 1 recovered and 0 were evicted. [ 2872.858120] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2872.872106] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2873.191878] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2873.309763] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2873.375699] Lustre: lustre-OST0001: new disk, initializing [ 2873.378382] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2873.382151] Lustre: 106026:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2873.663641] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2873.679735] Lustre: Skipped 2 previous similar messages [ 2879.970594] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2883.694798] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2883.707658] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2883.842590] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2888.214832] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2898.688107] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2904.548113] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2904.554226] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2904.577104] Lustre: Skipped 2 previous similar messages [ 2904.588180] Lustre: Skipped 1 previous similar message [ 2913.332623] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.52@tcp (stopping) [ 2913.336646] Lustre: Skipped 5 previous similar messages [ 2918.367143] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2918.642614] Lustre: server umount lustre-MDT0001 complete [ 2919.906859] LustreError: 103707:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2919.936744] LustreError: 103707:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2929.702651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2929.978115] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2929.984069] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2930.078978] Lustre: 103695:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2931.199512] LustreError: 107551:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2931.226278] LustreError: 107551:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2931.265151] LustreError: 103698:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2931.414107] LustreError: 107569:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.152@tcp: cfg command failed: rc = -17 [ 2931.419903] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2931.419903] [ 2936.228633] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2936.305673] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2948.093051] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 13:00:16 (1788800416) [ 2966.132604] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 2984.491045] Lustre: 109642:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3006.921857] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3010.313125] Lustre: 110790:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3089.376565] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3089.388821] LustreError: Skipped 1 previous similar message [ 3089.392379] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3089.406343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3089.410260] Lustre: Skipped 4 previous similar messages [ 3094.518139] Lustre: server umount lustre-MDT0000 complete [ 3095.017561] LustreError: 103701:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3095.032864] LustreError: 103701:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 3097.881635] LustreError: 103687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788800567 with bad export cookie 16646325704053708726 [ 3097.884607] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3097.902607] LustreError: 103687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3098.269449] Lustre: server umount lustre-MDT0001 complete [ 3112.670064] Lustre: server umount lustre-OST0000 complete [ 3126.569231] Lustre: server umount lustre-OST0001 complete [ 3131.997528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3144.469025] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3154.481560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3165.127842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3184.258238] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 3194.445955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3194.476445] Lustre: lustre-MDT0000: reset Object Index mappings [ 3195.065135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3195.069656] Lustre: Skipped 1 previous similar message [ 3199.379235] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3207.343823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3207.357935] Lustre: lustre-MDT0001: reset Object Index mappings [ 3211.914076] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3214.782810] Lustre: 115882:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3221.951164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3229.832957] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3232.191135] LustreError: 116234:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3232.226109] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 3232.233600] LustreError: 116234:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 3239.359615] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3239.419843] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:382 to 0x280000401:418) [ 3245.577693] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:199 to 0x2c0000400:225) [ 3245.601702] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3246.647525] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3254.827874] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3258.549600] Lustre: 117761:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3301.248230] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 13:06:10 (1788800770) [ 3327.470115] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3327.482510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3327.505666] Lustre: Skipped 5 previous similar messages [ 3327.515037] Lustre: Skipped 4 previous similar messages [ 3333.918510] Lustre: server umount lustre-MDT0000 complete [ 3338.210095] LustreError: 115486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788800808 with bad export cookie 16646325704053755801 [ 3338.212279] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3338.227262] LustreError: 115486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3344.817542] Lustre: server umount lustre-MDT0001 complete [ 3356.139496] Lustre: server umount lustre-OST0000 complete [ 3366.690989] Lustre: server umount lustre-OST0001 complete [ 3374.184728] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 3386.238565] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 3427.289034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3427.579866] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3427.607168] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3427.687715] Lustre: lustre-MDT0000: new disk, initializing [ 3427.801702] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3427.811394] Lustre: Skipped 3 previous similar messages [ 3427.839514] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3433.112381] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3439.738282] Lustre: server umount lustre-MDT0000 complete [ 3449.282305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3449.610784] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3449.627655] Lustre: 122031:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3449.646699] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3454.545124] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3459.714487] Lustre: server umount lustre-MDT0000 complete [ 3469.352237] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 13:08:58 (1788800938) [ 3471.527121] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3473.483293] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 13:09:02 (1788800942) [ 3494.876980] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 3508.217748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3514.849653] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3523.933601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3524.051610] Lustre: 124081:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3524.132563] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3524.138579] Lustre: Skipped 1 previous similar message [ 3524.314321] Lustre: lustre-MDT0001: new disk, initializing [ 3524.498055] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3524.520204] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3529.141289] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3534.426431] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3541.112834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3541.514660] Lustre: lustre-OST0000: new disk, initializing [ 3541.517457] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3541.526827] Lustre: 125662:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3543.405201] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3543.442989] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3543.566262] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3547.261812] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3558.210809] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3558.396845] Lustre: lustre-OST0001: new disk, initializing [ 3558.401376] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3558.414798] Lustre: 126531:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3560.605694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3560.622852] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3560.783708] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3566.106479] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3577.301885] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3580.787570] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3918.224171] Lustre: 132937:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3918.230338] Lustre: 132937:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3918.237375] Lustre: 132937:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3918.247861] Lustre: 132937:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3918.254692] Lustre: 132937:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3918.261253] Lustre: 132937:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4187.663665] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 13:20:56 (1788801656) [ 4205.554702] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4205.573285] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4205.601359] Lustre: Skipped 3 previous similar messages [ 4205.613227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4205.631824] Lustre: Skipped 10 previous similar messages [ 4206.063087] LustreError: 124095:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4206.104686] LustreError: 124095:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 14 previous similar messages [ 4206.211432] Lustre: server umount lustre-MDT0000 complete [ 4210.752297] LustreError: 124074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788801680 with bad export cookie 16646325704053820474 [ 4210.762718] LustreError: 124074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4210.769262] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4211.169769] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4211.195637] Lustre: Skipped 4 previous similar messages [ 4215.212682] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4215.227614] Lustre: Skipped 3 previous similar messages [ 4217.586608] Lustre: server umount lustre-MDT0001 complete [ 4229.041618] Lustre: server umount lustre-OST0000 complete [ 4241.741535] Lustre: server umount lustre-OST0001 complete [ 4250.948361] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 4262.868794] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 4304.570780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4304.977681] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4305.024619] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4305.140667] Lustre: lustre-MDT0000: new disk, initializing [ 4305.311234] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4305.326408] Lustre: Skipped 5 previous similar messages [ 4305.366814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4310.836486] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4321.890029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4322.015568] Lustre: 135990:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4322.069046] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4322.073884] Lustre: Skipped 1 previous similar message [ 4322.149488] Lustre: lustre-MDT0001: new disk, initializing [ 4322.294369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4322.310072] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4326.737806] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4336.855870] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4343.676727] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4351.154229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4351.514562] Lustre: lustre-OST0000: new disk, initializing [ 4351.520808] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4351.540455] Lustre: 137958:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4351.655867] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4351.661305] Lustre: Skipped 1 previous similar message [ 4352.724895] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4352.748786] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4352.825647] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4357.969183] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4367.922422] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4377.699221] Lustre: lustre-MDD0000: changelog on [ 4380.234962] Lustre: lustre-MDD0001: changelog on [ 6080.656985] Lustre: lustre-MDD0001: changelog off [ 6082.639635] Lustre: 139753:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6083.489343] Lustre: lustre-MDD0000: changelog off [ 6091.759343] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6091.760478] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6091.770034] Lustre: Skipped 1 previous similar message [ 6095.831379] Lustre: server umount lustre-OST0000 complete [ 6101.986244] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6101.990416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6102.014573] Lustre: Skipped 1 previous similar message [ 6102.031120] Lustre: Skipped 2 previous similar messages [ 6106.017392] Lustre: server umount lustre-MDT0000 complete [ 6107.105435] LustreError: 136881:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6107.120656] LustreError: 136881:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 6109.687556] LustreError: 146121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788803579 with bad export cookie 16646325704054639600 [ 6109.688720] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6109.701662] LustreError: 146121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6110.101102] Lustre: server umount lustre-MDT0001 complete [ 6120.854518] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 6124.036892] Key type lgssc unregistered [ 6124.389292] LNet: 147074:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6124.402806] LNetError: 147074:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6125.477709] LNet: Removed LNI 192.168.201.152@tcp [ 6127.061827] Key type .llcrypt unregistered [ 6127.064851] Key type ._llcrypt unregistered [ 6148.633809] Key type ._llcrypt registered [ 6148.635525] Key type .llcrypt registered [ 6148.801350] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 6167.657852] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 6169.485260] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6169.540451] alg: No test for adler32 (adler32-zlib) [ 6170.746617] Lustre: Lustre: Build Version: 2.17.58_39_gc924a9e [ 6171.189837] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 6172.943239] Key type lgssc registered [ 6174.503375] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6217.753193] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6217.798580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6219.187223] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6219.221888] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6219.338710] Lustre: lustre-MDT0000: new disk, initializing [ 6219.452799] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6219.480525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6225.511458] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6237.872636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6238.102788] Lustre: 150278:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6238.226701] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6238.233719] Lustre: Skipped 1 previous similar message [ 6238.357908] Lustre: lustre-MDT0001: new disk, initializing [ 6238.463185] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6238.511034] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6238.547902] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6244.526752] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6254.083530] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6260.035553] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6267.736558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6268.065429] Lustre: lustre-OST0000: new disk, initializing [ 6268.071395] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6268.083545] Lustre: 152246:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6268.244264] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6275.139891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6275.147681] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6275.271102] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6275.321104] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6286.316937] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6292.205339] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_module obdecho/obdecho [ 6294.477461] LustreError: 153248:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6294.482621] LustreError: 153248:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6296.441768] LustreError: 153344:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6310.375848] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6310.389923] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6310.412479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6311.395523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6311.407909] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6313.487361] Lustre: server umount lustre-MDT0000 complete [ 6318.065047] LustreError: 150269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788803788 with bad export cookie 10963081859000406892 [ 6318.083610] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6318.085549] LustreError: 150269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6318.413125] Lustre: server umount lustre-MDT0001 complete [ 6333.317105] Lustre: server umount lustre-OST0000 complete [ 6355.394592] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 6367.166297] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 6402.833590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6403.133615] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6403.198406] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6403.300768] Lustre: lustre-MDT0000: new disk, initializing [ 6403.454206] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6403.482989] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6408.411921] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6421.366350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6421.561631] Lustre: 157060:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6421.704411] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6421.714321] Lustre: Skipped 1 previous similar message [ 6421.847239] Lustre: lustre-MDT0001: new disk, initializing [ 6422.030092] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6422.104109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6422.116500] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6426.738187] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6437.004535] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6443.532407] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6450.656928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6450.966412] Lustre: lustre-OST0000: new disk, initializing [ 6450.969965] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6450.975571] Lustre: 159029:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6451.058775] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6458.308726] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6458.459055] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6458.466256] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6458.566474] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6467.688291] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6473.018223] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6473.221916] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6478.067130] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6478.252264] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6484.449578] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6484.463095] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6484.486075] Lustre: Skipped 1 previous similar message [ 6484.494156] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6484.506023] Lustre: Skipped 3 previous similar messages [ 6486.437452] Lustre: server umount lustre-OST0000 complete [ 6493.664254] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6493.680157] LustreError: Skipped 1 previous similar message [ 6493.687833] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6493.704642] Lustre: Skipped 1 previous similar message [ 6493.710802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6496.954610] Lustre: server umount lustre-MDT0000 complete [ 6499.814258] LustreError: 159036:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6499.834906] LustreError: 159036:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 6501.104091] LustreError: 157855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788803971 with bad export cookie 10963081859000407858 [ 6501.108550] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6501.115101] LustreError: 157855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6501.534753] Lustre: server umount lustre-MDT0001 complete [ 6505.611509] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 13:59:34 (1788803974) [ 6507.430876] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6509.124215] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 13:59:38 (1788803978) [ 6516.869652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6517.252255] LustreError: 161442:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6517.372328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6522.163816] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6522.848498] LustreError: 161443:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6527.975795] LustreError: 161442:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6532.535787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6539.122039] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6550.713250] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6557.943228] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6564.882416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6565.283454] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6565.287437] Lustre: Skipped 1 previous similar message [ 6572.635305] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6580.744902] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6599.141912] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6599.153790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6599.168096] Lustre: Skipped 3 previous similar messages [ 6599.194268] Lustre: Skipped 4 previous similar messages [ 6603.122596] Lustre: server umount lustre-MDT0000 complete [ 6604.266805] LustreError: 162170:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6604.297352] LustreError: 162170:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6607.221740] LustreError: 161423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788804077 with bad export cookie 10963081859000408740 [ 6607.233531] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6607.252664] LustreError: 161423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6609.378371] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6609.390246] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6609.393706] Lustre: Skipped 1 previous similar message [ 6612.385984] LustreError: 161439:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6612.425116] LustreError: 161439:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 6613.626875] Lustre: server umount lustre-MDT0001 complete [ 6621.077482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6635.554544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6635.775116] LustreError: 165318:0:(llog.c:1655:llog_backup()) MGC192.168.201.152@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6635.781864] Lustre: 165318:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.152@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6637.027193] LustreError: 165323:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6672.427749] LustreError: 165323:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6672.447600] LustreError: 165323:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 25 previous similar messages [ 6696.421119] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6696.426070] Lustre: 165318:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.152@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6737.406820] LustreError: 165325:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6737.413648] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6737.433962] LustreError: 165325:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 41 previous similar messages [ 6742.502268] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6742.847225] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6752.608196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6768.108170] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 6777.319058] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x9824b0dda93fb3e1 [ 6777.346136] LustreError: 166075:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.201.152@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (000000008cb804f7) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 6777.367548] Lustre: MGC192.168.201.152@tcp: Connection restored to 0@lo (at 0@lo) [ 6782.972200] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6784.813190] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6794.397128] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6796.309797] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6803.537151] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6805.246162] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6828.393152] Lustre: server umount lustre-MDT0000 complete [ 6829.025761] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6829.028083] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6829.071034] Lustre: Skipped 1 previous similar message [ 6833.294517] LustreError: 161424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788804303 with bad export cookie 10963081859000415201 [ 6833.307352] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6833.321436] LustreError: 161424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6833.859822] Lustre: server umount lustre-MDT0001 complete [ 6849.079609] Lustre: server umount lustre-OST0000 complete [ 6875.127100] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 6887.393514] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 6933.696464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6934.055948] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6934.174100] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6934.263735] Lustre: lustre-MDT0000: new disk, initializing [ 6934.432797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6934.436023] Lustre: Skipped 1 previous similar message [ 6934.528288] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6940.855409] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6956.127799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6956.281289] Lustre: 170434:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6956.505427] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6962.287822] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6972.213705] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6979.162306] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6986.549363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6986.816029] Lustre: lustre-OST0000: new disk, initializing [ 6986.818850] Lustre: Skipped 1 previous similar message [ 6986.822709] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6986.827367] Lustre: Skipped 2 previous similar messages [ 6986.832066] Lustre: 172403:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6993.864671] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6995.042307] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6995.050437] Lustre: Skipped 1 previous similar message [ 6995.059678] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6995.192285] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7003.027375] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7008.734622] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7008.956784] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7015.176504] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7015.419367] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7021.039206] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7021.047675] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7021.049609] LustreError: Skipped 1 previous similar message [ 7021.053242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7021.053250] Lustre: Skipped 2 previous similar messages [ 7021.092934] Lustre: Skipped 2 previous similar messages [ 7024.162363] Lustre: server umount lustre-OST0000 complete [ 7034.625312] Lustre: server umount lustre-MDT0000 complete [ 7036.387753] LustreError: 171515:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7036.400212] LustreError: 171515:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 38 previous similar messages [ 7039.173835] LustreError: 170425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788804509 with bad export cookie 10963081859000432036 [ 7039.178759] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7039.187753] LustreError: 170425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7039.460370] Lustre: server umount lustre-MDT0001 complete [ 7048.323321] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 14:08:37 (1788804517) [ 7056.054355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7060.976720] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7071.042462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7077.621818] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7085.541970] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7091.580324] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7098.653459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7106.265566] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7115.741750] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7132.051604] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 14:10:00 (1788804600) [ 7138.277885] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7138.298900] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7138.301912] Lustre: Skipped 2 previous similar messages [ 7138.313952] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7138.325357] LustreError: Skipped 2 previous similar messages [ 7138.332692] Lustre: Skipped 5 previous similar messages [ 7143.620774] Lustre: server umount lustre-OST0000 complete [ 7154.213221] Lustre: server umount lustre-MDT0000 complete [ 7158.309526] LustreError: 174795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788804628 with bad export cookie 10963081859000432918 [ 7158.335289] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7158.340776] LustreError: 174795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7158.979138] Lustre: server umount lustre-MDT0001 complete [ 7173.643719] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 7178.536307] Key type lgssc unregistered [ 7179.089641] LNet: 178548:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7179.112739] LNetError: 178548:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7179.149583] LNet: Removed LNI 192.168.201.152@tcp [ 7180.353708] Key type .llcrypt unregistered [ 7180.356979] Key type ._llcrypt unregistered [ 7204.660589] Key type ._llcrypt registered [ 7204.662681] Key type .llcrypt registered [ 7204.778401] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 7205.980074] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7206.010186] alg: No test for adler32 (adler32-zlib) [ 7207.514868] Lustre: Lustre: Build Version: 2.17.58_39_gc924a9e [ 7207.743495] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 7209.528551] Key type lgssc registered [ 7211.399587] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7221.049969] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7221.070500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7236.767421] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7241.888199] LustreError: 179792:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.152@tcp: failed processing log, type 4: rc = -110 [ 7247.075903] LustreError: 179792:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7247.097576] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7247.105278] LustreError: MGC192.168.201.152@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7247.122954] LustreError: 179792:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7247.142246] LustreError: 179792:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -22 [ 7247.154852] LustreError: 179792:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 7247.263525] Lustre: server umount lustre-OST0000 complete [ 7247.277848] LustreError: 179792:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7259.435657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7261.383738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7268.225888] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7271.907783] LustreError: 180292:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7271.934127] LustreError: 180292:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7277.025965] LustreError: 180293:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7279.365956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7280.084466] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7286.250498] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7297.309968] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7305.784511] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7314.500486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7314.735874] LustreError: 182109:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7314.744330] Lustre: 182109:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.201.152@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7314.916955] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7322.365663] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7330.835950] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7336.421264] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7336.428507] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7336.443353] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7336.455574] Lustre: Skipped 1 previous similar message [ 7336.476782] Lustre: Skipped 2 previous similar messages [ 7340.052171] Lustre: server umount lustre-MDT0000 complete [ 7341.539347] LustreError: 180293:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7341.552960] LustreError: 180293:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7345.048760] LustreError: 180272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788804815 with bad export cookie 16366470941247808030 [ 7345.055808] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7345.062353] LustreError: 180272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7345.952988] Lustre: server umount lustre-MDT0001 complete [ 7360.271617] Lustre: server umount lustre-OST0000 complete [ 7372.238954] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 7383.519639] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 7431.394523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7431.529847] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7431.550520] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7431.596242] Lustre: lustre-MDT0000: new disk, initializing [ 7431.656673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7431.673894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7437.509735] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7449.407233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7449.569509] Lustre: 185639:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7449.634465] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7449.638308] Lustre: Skipped 1 previous similar message [ 7449.751320] Lustre: lustre-MDT0001: new disk, initializing [ 7449.958271] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7450.044873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7450.065193] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7454.812772] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7465.349841] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7472.037714] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7478.819603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7479.139112] Lustre: lustre-OST0000: new disk, initializing [ 7479.143286] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7479.149848] Lustre: 187606:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7479.245605] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7484.515195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7484.521795] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7484.622454] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7485.867266] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7495.655273] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7501.402251] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7501.662693] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7506.079789] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7506.271404] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7510.499744] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7510.515090] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7510.546060] Lustre: Skipped 1 previous similar message [ 7510.553956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7514.323534] Lustre: server umount lustre-OST0000 complete [ 7520.738165] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7520.755824] Lustre: Skipped 1 previous similar message [ 7520.769725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7520.772448] Lustre: Skipped 2 previous similar messages [ 7524.642382] Lustre: server umount lustre-MDT0000 complete [ 7525.856263] LustreError: 185650:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7525.872955] LustreError: 185650:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7528.467033] LustreError: 185631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788804998 with bad export cookie 16366470941247808751 [ 7528.476060] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7528.488566] LustreError: 185631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7528.927958] Lustre: server umount lustre-MDT0001 complete [ 7537.697390] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 14:16:46 (1788805006) [ 7554.741439] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 7558.221701] Key type lgssc unregistered [ 7558.638404] LNet: 190551:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7558.658600] LNetError: 190551:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7558.683771] LNet: Removed LNI 192.168.201.152@tcp [ 7559.342391] Key type .llcrypt unregistered [ 7559.344374] Key type ._llcrypt unregistered [ 7576.447934] Key type ._llcrypt registered [ 7576.451380] Key type .llcrypt registered [ 7576.586826] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 7577.939885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7577.978695] alg: No test for adler32 (adler32-zlib) [ 7579.045799] Lustre: Lustre: Build Version: 2.17.58_39_gc924a9e [ 7579.286707] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 7580.975476] Key type lgssc registered [ 7582.375129] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7590.794854] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7590.826168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7592.422442] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7597.945278] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7602.658011] LustreError: 191767:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7602.680518] LustreError: 191767:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7607.780370] LustreError: 191766:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7607.914355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7608.498855] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7613.823321] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7621.165706] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7626.801331] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7634.871363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7635.071356] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7635.074798] LustreError: 193632:0:(llog.c:1696:llog_backup()) MGC192.168.201.152@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7635.081559] Lustre: 193632:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.201.152@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7635.210912] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7642.191434] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7650.117301] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7654.049721] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7654.053976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7654.082523] Lustre: Skipped 2 previous similar messages [ 7654.109328] Lustre: Skipped 2 previous similar messages [ 7659.498283] LustreError: 191767:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7659.521436] LustreError: 191767:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7659.589633] Lustre: server umount lustre-MDT0000 complete [ 7663.552986] LustreError: 191747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788805133 with bad export cookie 14351688734100841622 [ 7663.576650] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7663.577834] LustreError: 191747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7664.351555] LustreError: 192497:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000084bb22c9 x1875698043090048/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7664.567222] Lustre: server umount lustre-MDT0001 complete [ 7678.929575] Lustre: server umount lustre-OST0000 complete [ 7689.231408] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 7700.920140] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 7739.661951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7739.907044] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7739.934643] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7740.024508] Lustre: lustre-MDT0000: new disk, initializing [ 7740.074748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7740.093728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7744.277886] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7755.497158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7755.653414] Lustre: 197155:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7755.731305] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7755.735298] Lustre: Skipped 1 previous similar message [ 7755.737740] LustreError: 197940:0:(llog.c:1696:llog_backup()) MGC192.168.201.152@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7755.750061] LustreError: 197940:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7755.753978] Lustre: 197940:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.152@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7755.774900] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7755.782750] Lustre: Skipped 1 previous similar message [ 7755.908738] Lustre: lustre-MDT0001: new disk, initializing [ 7756.081305] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7756.128039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7756.144042] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7761.568185] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7771.319876] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7777.833511] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7785.606329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7785.897735] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7785.899674] Lustre: Skipped 1 previous similar message [ 7785.911649] LustreError: 199103:0:(llog.c:1696:llog_backup()) MGC192.168.201.152@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7785.918778] LustreError: 199103:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7785.924282] Lustre: 199103:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.152@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7785.932509] Lustre: 199103:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7785.963953] Lustre: lustre-OST0000: new disk, initializing [ 7785.970800] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7785.977037] Lustre: 199126:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7786.085651] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7793.674188] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7794.274631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7794.288021] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7794.497540] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7803.924948] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7808.449228] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7808.644093] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7813.598943] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7813.783125] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7820.256605] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7820.262973] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7820.278638] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7820.286802] Lustre: Skipped 1 previous similar message [ 7820.304811] Lustre: Skipped 1 previous similar message [ 7822.141831] Lustre: server umount lustre-OST0000 complete [ 7827.945695] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7827.953501] LustreError: Skipped 1 previous similar message [ 7827.957143] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7827.978179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7830.497941] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7830.498748] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7830.537693] Lustre: Skipped 1 previous similar message [ 7832.598465] Lustre: server umount lustre-MDT0000 complete [ 7835.619689] LustreError: 197165:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7835.639695] LustreError: 197165:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7836.455432] LustreError: 197953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788805306 with bad export cookie 14351688734100842343 [ 7836.459904] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7836.464949] LustreError: 197953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7836.888671] Lustre: server umount lustre-MDT0001 complete [ 7845.266353] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 14:21:54 (1788805314) [ 7861.018464] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 7871.313137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7871.807365] LustreError: 202255:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7871.869599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7876.572979] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7881.191385] LustreError: 202256:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7881.218037] LustreError: 202256:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7885.251553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7890.451222] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7893.748507] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7900.614904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7907.711502] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7917.351300] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7917.616574] Lustre: lustre-OST0001: new disk, initializing [ 7917.620112] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7917.625228] Lustre: 204560:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7917.769064] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7917.775033] Lustre: Skipped 2 previous similar messages [ 7924.640243] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7925.929785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7925.955678] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7926.065064] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7934.807434] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7940.572303] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7942.747778] Lustre: DEBUG MARKER: ADD OST3 [ 7948.283502] loop: module loaded [ 7948.381357] loop0: detected capacity change from 0 to 204800000 [ 7948.827913] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7948.845293] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7948.873350] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7955.184754] loop0: detected capacity change from 0 to 204800000 [ 7956.144388] Lustre: DEBUG MARKER: STOP OST3 [ 7961.115396] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7961.291047] Lustre: lustre-OST0003: new disk, initializing [ 7961.298616] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7961.306122] Lustre: 206568:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7963.086656] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7963.261911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7963.270622] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7963.330653] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7964.879362] Lustre: Failing over lustre-OST0003 [ 7965.010826] Lustre: server umount lustre-OST0003 complete [ 7965.092125] LustreError: 206746:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7967.190629] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7967.801546] LustreError: 203740:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.201.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7968.242558] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7968.256830] Lustre: Skipped 1 previous similar message [ 7975.812972] Lustre: DEBUG MARKER: START OST3 again [ 7982.776159] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7983.157955] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7983.170745] Lustre: Skipped 1 previous similar message [ 7983.206661] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7984.554319] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7984.701467] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 7984.702411] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7988.979921] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8008.818081] Lustre: Failing over lustre-OST0003 [ 8008.887231] Lustre: server umount lustre-OST0003 complete [ 8010.208478] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8010.217198] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8010.234144] LustreError: 204500:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8010.246309] LustreError: 204500:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 8029.153639] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8029.161867] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8029.163034] LustreError: Skipped 1 previous similar message [ 8029.166083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8029.171505] Lustre: Skipped 4 previous similar messages [ 8029.184887] Lustre: Skipped 3 previous similar messages [ 8032.837214] Lustre: server umount lustre-MDT0000 complete [ 8036.782740] LustreError: 202237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788805506 with bad export cookie 14351688734100843225 [ 8036.793452] LustreError: 202237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8036.794905] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8039.404321] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8039.433146] Lustre: Skipped 1 previous similar message [ 8051.167318] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8051.482421] Lustre: server umount lustre-MDT0001 complete [ 8055.132622] Lustre: server umount lustre-OST0000 complete [ 8058.627832] Lustre: server umount lustre-OST0001 complete [ 8064.244789] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 8070.823260] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 8102.308565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8102.503627] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8102.528712] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8102.615677] Lustre: lustre-MDT0000: new disk, initializing [ 8102.781363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8106.917599] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8116.522333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8116.619488] Lustre: 212018:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8116.817049] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8116.821945] Lustre: Skipped 1 previous similar message [ 8116.872355] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8121.688620] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8130.564293] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8135.741259] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8143.658478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8144.038514] Lustre: lustre-OST0000: new disk, initializing [ 8144.042056] Lustre: Skipped 1 previous similar message [ 8144.047839] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8144.053139] Lustre: Skipped 2 previous similar messages [ 8144.063511] Lustre: 213988:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8145.773880] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8145.782820] Lustre: Skipped 1 previous similar message [ 8145.793519] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8145.892261] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8150.244244] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8158.545591] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8163.330425] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8163.575848] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8167.861561] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8168.048730] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8171.498996] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8171.501161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8171.516799] Lustre: Skipped 3 previous similar messages [ 8171.518986] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8171.537357] Lustre: Skipped 5 previous similar messages [ 8175.884257] Lustre: server umount lustre-OST0000 complete [ 8185.946807] Lustre: server umount lustre-MDT0000 complete [ 8186.858419] LustreError: 212903:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8186.881491] LustreError: 212903:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 20 previous similar messages [ 8189.489788] LustreError: 212011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788805659 with bad export cookie 14351688734100845514 [ 8189.493439] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8189.497109] LustreError: 212011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8189.843878] Lustre: server umount lustre-MDT0001 complete [ 8193.127786] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 14:27:42 (1788805662) [ 8208.788881] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 8218.160108] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 8246.957215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8247.187612] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8247.211360] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8247.289711] Lustre: lustre-MDT0000: new disk, initializing [ 8247.360384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8250.835474] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8259.456804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8259.579253] Lustre: 218595:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8259.774555] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8263.206792] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8272.212446] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8278.439919] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8285.240462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8285.557536] Lustre: 220562:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8286.907639] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8290.681985] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8298.683407] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8302.029354] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8302.217477] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8306.267404] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8306.415845] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8312.291572] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8312.312563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8312.315817] Lustre: Skipped 3 previous similar messages [ 8312.339308] Lustre: Skipped 3 previous similar messages [ 8314.194574] Lustre: server umount lustre-OST0000 complete [ 8323.718851] Lustre: server umount lustre-MDT0000 complete [ 8327.270495] LustreError: 218588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788805797 with bad export cookie 14351688734100846403 [ 8327.283449] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8327.285739] LustreError: 218588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8327.538319] Lustre: server umount lustre-MDT0001 complete [ 8335.878634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8336.211884] LustreError: 222777:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8340.532868] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8349.155272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8353.693673] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8359.266038] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8363.158582] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8368.463630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8374.066500] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8379.487320] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8390.631344] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8390.644994] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8390.650669] Lustre: Skipped 4 previous similar messages [ 8394.458456] Lustre: server umount lustre-OST0000 complete [ 8403.939132] Lustre: server umount lustre-MDT0000 complete [ 8407.123335] LustreError: 222758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788805877 with bad export cookie 14351688734100847285 [ 8407.131699] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8407.522755] Lustre: server umount lustre-MDT0001 complete [ 8519.468324] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 8521.795592] Key type lgssc unregistered [ 8522.026409] LNet: 226442:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8522.035901] LNetError: 226442:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8522.060194] LNet: Removed LNI 192.168.201.152@tcp [ 8522.875151] Key type .llcrypt unregistered [ 8522.880140] Key type ._llcrypt unregistered [ 8544.253655] Key type ._llcrypt registered [ 8544.255902] Key type .llcrypt registered [ 8544.388527] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 8556.458707] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 8557.948139] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8557.980322] alg: No test for adler32 (adler32-zlib) [ 8559.002476] Lustre: Lustre: Build Version: 2.17.58_39_gc924a9e [ 8559.168388] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 8560.791752] Key type lgssc registered [ 8561.688842] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8587.734395] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8587.751395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8588.905399] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8588.950958] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8589.045712] Lustre: lustre-MDT0000: new disk, initializing [ 8589.090817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8589.104276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8592.608767] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8601.432425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8601.601449] Lustre: 229767:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8601.672456] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8601.675956] Lustre: Skipped 1 previous similar message [ 8601.792140] Lustre: lustre-MDT0001: new disk, initializing [ 8601.935399] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8601.980578] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8601.990447] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8605.594577] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8614.024490] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8618.745731] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8625.928134] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8626.158918] Lustre: lustre-OST0000: new disk, initializing [ 8626.162779] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8626.170041] Lustre: 231733:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8626.269738] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8631.686326] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8634.968245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8634.976736] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8635.083406] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8639.700214] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8643.713252] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8643.835096] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8649.094225] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8649.265852] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8655.850245] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8655.854982] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8655.860158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8655.877339] Lustre: Skipped 1 previous similar message [ 8657.739934] Lustre: server umount lustre-OST0000 complete [ 8663.520418] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8663.525815] LustreError: Skipped 1 previous similar message [ 8663.533889] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8663.545663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8663.558048] Lustre: Skipped 1 previous similar message [ 8666.084243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8666.090819] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8667.579704] Lustre: server umount lustre-MDT0000 complete [ 8671.182086] LustreError: 229761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788806141 with bad export cookie 7361865580635404777 [ 8671.184450] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8671.195934] LustreError: 229761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8671.502427] Lustre: server umount lustre-MDT0001 complete [ 8688.117682] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 8695.998220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8696.477261] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8699.789914] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8701.931613] LustreError: 234668:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8701.950189] LustreError: 234668:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8706.021021] LustreError: 234669:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8706.288514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8706.519024] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8709.989449] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8712.120138] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8717.624507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8717.916261] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8722.847901] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8729.986082] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8730.137983] Lustre: lustre-OST0001: new disk, initializing [ 8730.141279] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8730.150664] Lustre: 236972:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8735.613653] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8737.382994] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8737.391350] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8737.436620] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8745.210472] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8749.687620] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8758.049794] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 14:37:07 (1788806227) [ 8762.848293] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8762.855142] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8762.862653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8762.865202] Lustre: Skipped 1 previous similar message [ 8765.762316] Lustre: server umount lustre-MDT0000 complete [ 8768.483897] LustreError: 234668:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8768.500479] LustreError: 234668:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8768.664378] LustreError: 234649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788806238 with bad export cookie 7361865580635405659 [ 8768.666868] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8768.674416] LustreError: 234649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8768.909744] Lustre: server umount lustre-MDT0001 complete [ 8782.865222] Lustre: server umount lustre-OST0000 complete [ 8795.997316] Lustre: server umount lustre-OST0001 complete [ 8800.524254] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 8807.655456] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 8836.884759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8837.021182] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8837.039322] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8837.093341] Lustre: lustre-MDT0000: new disk, initializing [ 8837.144506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8837.149601] Lustre: Skipped 1 previous similar message [ 8837.162325] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8840.428532] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8849.327589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8849.490128] Lustre: 240916:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8849.547689] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8849.551739] Lustre: Skipped 1 previous similar message [ 8849.625719] Lustre: lustre-MDT0001: new disk, initializing [ 8849.779479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8849.791031] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8853.367289] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8860.457714] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8864.209224] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8868.983875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8869.178493] Lustre: lustre-OST0000: new disk, initializing [ 8869.181260] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8869.188059] Lustre: 242885:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8869.241269] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8869.245630] Lustre: Skipped 1 previous similar message [ 8870.590279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8870.609818] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8870.689189] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8873.834824] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8880.669679] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8884.069905] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8884.258810] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8887.977948] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8888.141929] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8891.362847] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8891.373414] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8891.382659] Lustre: Skipped 3 previous similar messages [ 8891.387984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8891.397515] Lustre: Skipped 5 previous similar messages [ 8895.860504] Lustre: server umount lustre-OST0000 complete [ 8901.088326] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8901.094306] LustreError: Skipped 1 previous similar message [ 8901.097804] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8901.111585] Lustre: Skipped 1 previous similar message [ 8901.117522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8904.802228] Lustre: server umount lustre-MDT0000 complete [ 8906.720829] LustreError: 240927:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8906.729817] LustreError: 240927:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8907.715256] LustreError: 240906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788806377 with bad export cookie 7361865580635407794 [ 8907.719667] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8907.721793] LustreError: 240906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8907.948294] Lustre: server umount lustre-MDT0001 complete [ 8914.005231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8914.266844] LustreError: 245098:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8917.404355] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8923.308431] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8927.018357] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8932.593208] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8938.065740] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8944.212539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8944.445568] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8944.453244] Lustre: Skipped 2 previous similar messages [ 8949.256756] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8954.858484] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9102.305899] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9102.310313] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9102.317150] Lustre: Skipped 1 previous similar message [ 9102.320944] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9102.334605] Lustre: Skipped 1 previous similar message [ 9114.079466] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9114.210684] Lustre: server umount lustre-OST0000 complete [ 9123.372664] Lustre: server umount lustre-MDT0000 complete [ 9123.809955] LustreError: 247597:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9123.835928] LustreError: 247597:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9126.062218] LustreError: 245078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788806596 with bad export cookie 7361865580635408676 [ 9126.065837] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9126.066949] LustreError: 245078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9126.334234] Lustre: server umount lustre-MDT0001 complete [ 9222.305623] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 9224.410648] Key type lgssc unregistered [ 9224.739126] LNet: 248700:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9224.744693] LNetError: 248700:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9224.754350] LNet: Removed LNI 192.168.201.152@tcp [ 9225.424146] Key type .llcrypt unregistered [ 9225.425386] Key type ._llcrypt unregistered [ 9241.765853] Key type ._llcrypt registered [ 9241.767562] Key type .llcrypt registered [ 9241.835791] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 9251.793957] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 9252.425832] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9252.453834] alg: No test for adler32 (adler32-zlib) [ 9253.369575] Lustre: Lustre: Build Version: 2.17.58_39_gc924a9e [ 9253.507882] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 9255.175250] Key type lgssc registered [ 9256.016254] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9282.546455] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 9288.279288] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9288.294653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9289.430449] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9289.450702] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9289.506359] Lustre: lustre-MDT0000: new disk, initializing [ 9289.545045] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9289.552472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9292.457705] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 9299.613444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9299.681301] Lustre: 252742:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9299.710698] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9299.712540] Lustre: Skipped 1 previous similar message [ 9299.758726] Lustre: lustre-MDT0001: new disk, initializing [ 9299.831799] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9299.855647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9299.863049] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9302.539417] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 9306.030960] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9310.794732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9310.981050] Lustre: lustre-OST0000: new disk, initializing [ 9310.985327] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9310.992149] Lustre: 254370:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9311.066740] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9311.672742] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9311.678658] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9311.802302] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9315.596546] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 9323.198427] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9323.333314] Lustre: lustre-OST0001: new disk, initializing [ 9323.335327] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9323.339460] Lustre: 255241:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9323.400712] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9327.864784] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 9329.191026] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9329.199296] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9329.253503] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9336.665818] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9347.240356] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9354.728916] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9354.732274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9354.737194] Lustre: Skipped 1 previous similar message [ 9359.717251] Lustre: server umount lustre-MDT0000 complete [ 9362.079259] LustreError: 253544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788806832 with bad export cookie 213323755128875844 [ 9362.089991] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9362.342408] Lustre: server umount lustre-MDT0001 complete [ 9374.231294] Lustre: server umount lustre-OST0000 complete [ 9385.152952] Lustre: server umount lustre-OST0001 complete [ 9392.768796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9397.944708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9402.791957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9408.344230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9419.064820] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 9424.693454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9424.705796] Lustre: lustre-MDT0000: reset Object Index mappings [ 9424.950773] LustreError: 260247:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9424.961642] LustreError: 260247:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 9425.014506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9427.408557] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 9430.499727] LustreError: 260248:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9432.558738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9432.569503] Lustre: lustre-MDT0001: reset Object Index mappings [ 9434.994718] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 9436.744784] Lustre: 261386:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9440.309424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9440.532694] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9440.536211] Lustre: Skipped 1 previous similar message [ 9442.602243] LustreError: 261739:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9443.472725] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 9447.697965] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9450.931947] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9450.934487] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9451.174141] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 9455.861549] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9457.980847] Lustre: 263257:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9461.219417] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9461.224286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9461.242332] Lustre: Skipped 3 previous similar messages [ 9461.247286] Lustre: Skipped 3 previous similar messages [ 9466.341295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9466.348203] Lustre: Skipped 6 previous similar messages [ 9466.691951] Lustre: server umount lustre-MDT0000 complete [ 9469.046252] LustreError: 260228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788806939 with bad export cookie 213323755128882634 [ 9469.046647] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9469.054478] LustreError: 260228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9469.267603] Lustre: server umount lustre-MDT0001 complete [ 9481.889580] Lustre: server umount lustre-OST0000 complete [ 9494.462980] Lustre: server umount lustre-OST0001 complete [ 9503.550835] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 14:49:33 (1788806973) [ 9512.188790] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 9516.746750] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 9534.774567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9534.898227] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9534.918135] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9534.979209] Lustre: lustre-MDT0000: new disk, initializing [ 9535.026266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9535.030052] Lustre: Skipped 1 previous similar message [ 9535.037940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9537.056860] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 9543.184394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9543.250104] Lustre: 267504:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9543.298704] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9543.301742] Lustre: Skipped 1 previous similar message [ 9543.351556] Lustre: lustre-MDT0001: new disk, initializing [ 9543.450993] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9543.459585] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9545.756733] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 9551.156258] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9554.119606] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9557.697472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9557.839980] Lustre: 269471:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9559.901645] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9559.906611] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9559.961711] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9560.881791] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 9566.109747] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9568.390825] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9568.533335] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9570.938885] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9571.065471] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9575.396919] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9575.397028] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9575.399779] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9575.420256] Lustre: Skipped 3 previous similar messages [ 9578.405316] Lustre: server umount lustre-OST0000 complete [ 9584.608787] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9584.613275] LustreError: Skipped 1 previous similar message [ 9584.615509] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9584.626170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9584.629547] Lustre: Skipped 1 previous similar message [ 9595.359142] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9595.434570] Lustre: server umount lustre-MDT0000 complete [ 9595.876874] LustreError: 267515:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9595.883703] LustreError: 267515:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9597.491328] LustreError: 267496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788807067 with bad export cookie 213323755128884727 [ 9597.497215] LustreError: 267496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9597.497466] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9597.690810] Lustre: server umount lustre-MDT0001 complete [ 9606.459796] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 9611.041902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9611.219607] LustreError: 272402:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9611.239897] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9611.242340] Lustre: Skipped 2 previous similar messages [ 9612.912200] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 9617.091942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9619.257802] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 9620.515873] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9623.446856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9626.432268] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 9631.407285] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9631.525792] Lustre: lustre-OST0001: new disk, initializing [ 9631.527721] Lustre: Skipped 1 previous similar message [ 9631.530446] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9631.532590] Lustre: Skipped 1 previous similar message [ 9631.536464] Lustre: 274708:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9633.375263] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9633.383910] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9633.421372] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9634.412662] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 9639.773788] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9641.547458] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9643.930286] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 9643.962469] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 9648.096399] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9648.100042] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9648.105655] Lustre: Skipped 1 previous similar message [ 9648.108534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9648.110729] Lustre: Skipped 4 previous similar messages [ 9653.738607] Lustre: server umount lustre-MDT0000 complete [ 9655.149238] LustreError: 272384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788807125 with bad export cookie 213323755128885609 [ 9655.152431] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9655.154064] LustreError: 272384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9655.285891] Lustre: server umount lustre-MDT0001 complete [ 9666.520569] Lustre: server umount lustre-OST0000 complete [ 9678.516864] Lustre: server umount lustre-OST0001 complete [ 9699.434699] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 9710.842425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9711.357757] LustreError: 278048:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9711.389136] LustreError: 278048:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9711.460288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9711.466798] Lustre: Skipped 3 previous similar messages [ 9717.195883] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 9725.617321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9730.283123] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 9733.153874] Lustre: 279189:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9739.865511] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9747.545771] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 9751.481568] LustreError: 279545:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9751.512137] LustreError: 279545:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9757.209759] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9757.706251] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 9762.845405] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 9767.765722] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 9779.114281] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9784.749301] Lustre: 281066:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9791.230172] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 9791.273216] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 9803.415493] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 14:54:32 (1788807272) [ 9818.082968] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9818.088326] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9818.099935] Lustre: Skipped 3 previous similar messages [ 9818.107485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9818.110265] Lustre: Skipped 4 previous similar messages [ 9824.225848] LustreError: 278048:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9824.232914] LustreError: 278048:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 9826.261903] Lustre: server umount lustre-MDT0000 complete [ 9829.750150] LustreError: 278029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788807299 with bad export cookie 213323755128891986 [ 9829.762725] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9834.466620] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9834.498573] Lustre: Skipped 4 previous similar messages [ 9844.191501] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9844.600160] Lustre: server umount lustre-MDT0001 complete [ 9854.271399] Lustre: server umount lustre-OST0000 complete [ 9857.658986] Lustre: server umount lustre-OST0001 complete [ 9863.315156] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 9871.293357] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 9903.726390] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 9912.010232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9912.264683] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9912.317989] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9912.402814] Lustre: lustre-MDT0000: new disk, initializing [ 9912.475454] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9912.482020] Lustre: Skipped 3 previous similar messages [ 9912.497740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9916.414751] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 9926.244977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9926.366805] Lustre: 285593:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9926.678191] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9931.501804] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 9942.397559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9942.676443] Lustre: 287225:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9944.847146] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9947.803438] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 9957.900478] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9958.123319] Lustre: 288096:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9959.506022] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9959.513115] Lustre: Skipped 1 previous similar message [ 9959.606649] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9964.205746] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 9973.156519] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9976.410712] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9976.414774] Lustre: Skipped 1 previous similar message [ 9988.006417] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 14:57:37 (1788807457) [ 9990.632431] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9990.641039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9990.643871] Lustre: Skipped 3 previous similar messages [ 9990.657983] Lustre: Skipped 13 previous similar messages [ 9996.172801] Lustre: server umount lustre-MDT0000 complete [ 9999.211945] LustreError: 285584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788807469 with bad export cookie 213323755128899945 [ 9999.225384] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9999.234708] LustreError: 285584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9999.489504] Lustre: server umount lustre-MDT0001 complete [10012.539081] Lustre: server umount lustre-OST0000 complete [10025.703425] Lustre: server umount lustre-OST0001 complete [10033.509125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10033.848916] LustreError: 290340:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10033.865850] LustreError: 290340:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [10037.507144] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [10045.869710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10051.207995] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [10058.658153] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10064.526032] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10070.740250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10076.776420] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [10083.543327] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10101.024080] loop1: detected capacity change from 0 to 204800000 [10101.270499] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10101.289748] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10101.307797] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10107.240484] loop1: detected capacity change from 0 to 204800000 [10113.083628] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10113.190478] LustreError: 290329:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs6442: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10113.200649] LustreError: 290329:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs6442 [10113.204853] LustreError: 290329:0:(mgs_handler.c:517:mgs_target_reg()) fs6442-MDT0001: Error checking target index: rc = -13 [10113.217467] LustreError: fs6442-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10113.225614] LustreError: 294055:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [10113.231272] LustreError: 294055:0:(tgt_mount.c:2128:server_put_super()) no obd fs6442-MDT0001 [10113.239485] LustreError: 294055:0:(tgt_mount.c:116:server_deregister_mount()) fs6442-MDT0001 not registered [10113.288358] Lustre: server umount fs6442-MDT0001 complete [10113.292507] LustreError: 294055:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10123.291666] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10123.496495] Lustre: Setting parameter fs6442-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs6442-MDT0001 [10123.601271] Lustre: srv-fs6442-MDT0001: No data found on store. Initialize space. [10123.613489] Lustre: Skipped 4 previous similar messages [10123.818381] Lustre: fs6442-MDT0001: new disk, initializing [10123.824164] Lustre: Skipped 3 previous similar messages [10124.007901] Lustre: srv-fs6442-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10124.021435] Lustre: 294521:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs6442-MDT0001-mdtlov: can't create new llog: rc = -115 [10124.029611] LustreError: 294521:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs6442-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10129.183930] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [10136.500966] Lustre: server umount fs6442-MDT0001 complete [10149.758321] loop1: detected capacity change from 0 to 204800000 [10155.505760] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10156.127223] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10156.701438] Lustre: 296090:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs6442-MDT0001-mdtlov: can't create new llog: rc = -115 [10156.707756] LustreError: 296090:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs6442-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10161.764659] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [10165.745942] LustreError: 292169:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10165.752660] LustreError: 292169:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 56 previous similar messages [10166.740049] Lustre: server umount fs6442-MDT0001 complete [10179.557821] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10179.569936] LustreError: Skipped 1 previous similar message [10179.577141] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10179.596778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10179.608629] Lustre: Skipped 4 previous similar messages [10183.891195] Lustre: server umount lustre-MDT0000 complete [10193.783277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10193.941876] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10194.355345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10194.370795] Lustre: Skipped 8 previous similar messages [10199.034206] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [10199.535285] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10207.380636] loop1: detected capacity change from 0 to 204800000 [10213.093088] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10213.784134] Lustre: 298549:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs6442-MDT0001-mdtlov: can't create new llog: rc = -115 [10213.792363] LustreError: 298549:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs6442-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10218.655751] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [10223.272708] Lustre: server umount fs6442-MDT0001 complete [10250.727623] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10251.989190] Lustre: server umount lustre-OST0000 complete [10262.185492] Lustre: server umount lustre-MDT0000 complete [10266.703793] LustreError: 290321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788807736 with bad export cookie 213323755128902626 [10266.722470] LustreError: 290321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10266.729462] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10267.272428] Lustre: server umount lustre-MDT0001 complete [10279.429443] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [10282.701480] Key type lgssc unregistered [10283.040868] LNet: 300895:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10283.048331] LNetError: 300895:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10283.064911] LNet: Removed LNI 192.168.201.152@tcp [10284.102221] Key type .llcrypt unregistered [10284.105620] Key type ._llcrypt unregistered [10306.149789] Key type ._llcrypt registered [10306.152409] Key type .llcrypt registered [10306.265736] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [10307.084214] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10307.116906] alg: No test for adler32 (adler32-zlib) [10308.346758] Lustre: Lustre: Build Version: 2.17.58_39_gc924a9e [10308.623571] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [10310.352721] Key type lgssc registered [10311.654698] Lustre: Echo OBD driver; http://www.lustre.org/ [10320.891748] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10320.943405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10322.571621] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10328.023872] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [10332.134242] LustreError: 302164:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10332.149078] LustreError: 302164:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10337.253449] LustreError: 302163:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10337.827112] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10338.565907] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10343.763973] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [10353.710383] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10360.163237] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10367.167317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10367.422978] LustreError: 303992:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10367.460464] LustreError: 303992:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10367.798861] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10373.094386] LustreError: 303994:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10374.228649] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [10378.222528] LustreError: 303993:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10382.226300] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10388.455311] LustreError: 303992:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10388.475883] LustreError: 303992:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [10402.578460] loop1: detected capacity change from 0 to 204800000 [10402.783675] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10402.793921] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10402.813904] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10405.433342] LustreError: 304003:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10405.452898] LustreError: 304003:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [10408.420819] loop1: detected capacity change from 0 to 204800000 [10414.700612] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10414.870384] Lustre: fs4005-OST0000: new disk, initializing [10414.874424] Lustre: srv-fs4005-OST0000: No data found on store. Initialize space. [10414.881431] Lustre: 305898:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10414.968344] Lustre: fs4005-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10422.274248] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [10430.578707] Lustre: server umount fs4005-OST0000 complete [10439.653262] LustreError: 303992:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10439.687907] LustreError: 303992:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 19 previous similar messages [10442.800994] loop1: detected capacity change from 0 to 204800000 [10448.130562] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10448.338598] Lustre: fs4005-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10449.897451] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10454.823927] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [10463.160134] loop2: detected capacity change from 0 to 204800000 [10463.451589] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10463.463277] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10463.472535] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10465.260236] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10468.714697] loop2: detected capacity change from 0 to 204800000 [10473.737775] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10473.818084] LustreError: fs4005-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [10473.832477] LustreError: fs4005-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10473.842617] LustreError: 308938:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [10473.857798] LustreError: 308938:0:(tgt_mount.c:2128:server_put_super()) no obd fs4005-OST0001 [10473.870249] LustreError: 308938:0:(tgt_mount.c:116:server_deregister_mount()) fs4005-OST0001 not registered [10473.936607] Lustre: server umount fs4005-OST0001 complete [10473.941947] LustreError: 308938:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10483.956374] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10484.143417] Lustre: fs4005-OST0001: new disk, initializing [10484.149833] Lustre: srv-fs4005-OST0001: No data found on store. Initialize space. [10484.171919] Lustre: 309489:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10484.263526] Lustre: fs4005-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10490.475619] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [10499.045844] Lustre: server umount fs4005-OST0000 complete [10506.229869] LustreError: 303992:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10506.249296] LustreError: 303992:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 38 previous similar messages [10508.147206] Lustre: server umount fs4005-OST0001 complete [10519.485165] loop1: detected capacity change from 0 to 204800000 [10524.229332] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10524.461607] Lustre: fs4005-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10531.480094] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [10537.922627] loop2: detected capacity change from 0 to 204800000 [10542.940716] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10548.922722] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [10557.441571] Lustre: server umount fs4005-OST0000 complete [10565.926971] Lustre: server umount fs4005-OST0001 complete [10577.896449] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10577.905063] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10577.910748] LustreError: Skipped 1 previous similar message [10577.913412] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10577.922962] Lustre: Skipped 1 previous similar message [10577.935707] Lustre: Skipped 1 previous similar message [10582.887485] Lustre: server umount lustre-OST0000 complete [10588.132373] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10588.134136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10588.159507] Lustre: Skipped 1 previous similar message [10592.861262] Lustre: server umount lustre-MDT0000 complete [10596.133198] LustreError: 302145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788808066 with bad export cookie 453777727079976214 [10596.148285] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10596.467655] Lustre: server umount lustre-MDT0001 complete [10605.502415] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [10607.888358] Key type lgssc unregistered [10608.146714] LNet: 315413:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10608.150718] LNetError: 315413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10609.192932] LNet: Removed LNI 192.168.201.152@tcp [10609.930392] Key type .llcrypt unregistered [10609.936096] Key type ._llcrypt unregistered [10639.994367] Key type ._llcrypt registered [10639.996813] Key type .llcrypt registered [10640.121624] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [10654.146288] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [10655.573652] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10655.594490] alg: No test for adler32 (adler32-zlib) [10656.583461] Lustre: Lustre: Build Version: 2.17.58_39_gc924a9e [10656.745304] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [10658.423312] Key type lgssc registered [10659.762691] Lustre: Echo OBD driver; http://www.lustre.org/ [10694.459864] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10694.474587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10695.745121] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10695.767844] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10695.873401] Lustre: lustre-MDT0000: new disk, initializing [10695.965365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10695.986446] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10700.225228] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [10709.543529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10709.633771] Lustre: 319269:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10709.681871] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10709.686076] Lustre: Skipped 1 previous similar message [10709.752866] Lustre: lustre-MDT0001: new disk, initializing [10709.844477] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10709.890699] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10709.895917] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10713.434646] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [10722.145804] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10728.040587] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10734.601202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10734.836304] Lustre: lustre-OST0000: new disk, initializing [10734.839556] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10734.846626] Lustre: 321234:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10734.971221] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10741.487499] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [10742.852273] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10742.866661] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10742.993040] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10751.161526] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10753.108040] LustreError: 317442:0:(lmv_obd.c:211:lmv_notify()) activation of lustre-MDT0000_UUID failed: -22 [10753.168439] Lustre: Mounted lustre-client - version 2.17.58_39_gc924a9e [10754.015513] Lustre: Unmounted lustre-client [10763.755165] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10763.755573] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10763.760941] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10763.772587] LustreError: Skipped 1 previous similar message [10763.801241] Lustre: Skipped 1 previous similar message [10768.140985] Lustre: server umount lustre-OST0000 complete [10771.426622] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10771.434298] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10771.469944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10771.474776] Lustre: Skipped 1 previous similar message [10773.992560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10773.992628] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10777.624701] Lustre: server umount lustre-MDT0000 complete [10781.292686] LustreError: 319261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788808251 with bad export cookie 16552259114157643578 [10781.300889] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10781.306281] LustreError: 319261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10781.852953] Lustre: server umount lustre-MDT0001 complete [10791.529446] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [10794.163132] Key type lgssc unregistered [10812.383411] LNet: 1 local NIs in recovery (showing 1): 192.168.201.152@tcp [10812.899777] LNet: 323394:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10812.914108] LNetError: 323394:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10884.650775] LNet: Removed LNI 192.168.201.152@tcp [10885.675600] Key type .llcrypt unregistered [10885.678639] Key type ._llcrypt unregistered [10902.740449] Key type ._llcrypt registered [10902.742723] Key type .llcrypt registered [10902.878634] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [10917.105808] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [10918.304964] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10918.394322] alg: No test for adler32 (adler32-zlib) [10919.459285] Lustre: Lustre: Build Version: 2.17.58_39_gc924a9e [10919.645884] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [10921.319245] Key type lgssc registered [10922.272809] Lustre: Echo OBD driver; http://www.lustre.org/ [10948.666440] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 15:13:37 (1788808417) [10954.441685] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10954.472197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10955.755366] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10955.794764] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10955.856721] Lustre: lustre-MDT0000: new disk, initializing [10955.916306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10955.930751] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10959.482981] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [10968.234852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10968.348512] Lustre: 326615:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10968.423259] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10968.438611] Lustre: Skipped 1 previous similar message [10968.570873] Lustre: lustre-MDT0001: new disk, initializing [10968.698197] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10968.739433] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10968.751920] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10972.736175] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [10980.329958] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10985.086278] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10990.318751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10990.515494] Lustre: lustre-OST0000: new disk, initializing [10990.518692] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10990.523776] Lustre: 328583:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10990.577908] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10995.055643] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [10998.861861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10998.877580] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10998.975764] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11002.844423] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11019.746145] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11019.746564] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11019.754171] LustreError: Skipped 1 previous similar message [11019.755970] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11019.789513] Lustre: Skipped 1 previous similar message [11020.879725] Lustre: server umount lustre-OST0000 complete [11024.866027] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11024.871038] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11024.876824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11024.885847] Lustre: Skipped 2 previous similar messages [11029.779059] Lustre: server umount lustre-MDT0000 complete [11033.181134] LustreError: 326607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788808503 with bad export cookie 1554489150585088421 [11033.184101] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11033.188765] LustreError: 326607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11033.478536] Lustre: server umount lustre-MDT0001 complete [11041.352189] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [11043.291621] Key type lgssc unregistered [11043.543722] LNet: 330489:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11043.556857] LNetError: 330489:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11043.571045] LNet: Removed LNI 192.168.201.152@tcp [11044.262466] Key type .llcrypt unregistered [11044.264263] Key type ._llcrypt unregistered [11059.940983] Key type ._llcrypt registered [11059.943268] Key type .llcrypt registered [11060.019917] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [11060.804448] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11060.962115] alg: No test for adler32 (adler32-zlib) [11062.022573] Lustre: Lustre: Build Version: 2.17.58_39_gc924a9e [11062.229555] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [11063.839974] Key type lgssc registered [11064.613546] Lustre: Echo OBD driver; http://www.lustre.org/ [11069.519374] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11069.531254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11070.802118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11073.854399] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [11080.164550] LustreError: 331738:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11080.172545] LustreError: 331738:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11080.624132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11080.922128] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11084.085274] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [11089.978550] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11094.996273] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11100.439785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11100.807911] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11105.860531] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [11111.364560] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11124.870880] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11125.962253] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11130.849670] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11130.856280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11130.859411] Lustre: Skipped 1 previous similar message [11130.867468] Lustre: Skipped 1 previous similar message [11133.504457] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.52@tcp (stopping) [11133.511662] Lustre: Skipped 1 previous similar message [11135.969761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11138.604975] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.52@tcp (stopping) [11138.614309] Lustre: Skipped 2 previous similar messages [11143.135138] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11143.265963] Lustre: server umount lustre-MDT0000 complete [11143.722529] LustreError: 331733:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11145.872489] LustreError: 331720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788808615 with bad export cookie 3015845090717231904 [11145.873176] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11145.882049] LustreError: 331720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11146.208050] LustreError: 331735:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11146.210453] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11146.233072] Lustre: Skipped 1 previous similar message [11146.236291] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11146.239763] Lustre: Skipped 3 previous similar messages [11148.840622] LustreError: 331733:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11152.131486] Lustre: server umount lustre-MDT0001 complete [11154.998691] Lustre: server umount lustre-OST0000 complete [11160.932858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11164.691436] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [11166.828797] Lustre: 335916:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.152@tcp [11168.411403] Lustre: server umount MGS complete [11174.375787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11174.719734] LustreError: 336487:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11174.735764] LustreError: 336487:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11174.787065] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11178.507505] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [11184.100569] LustreError: 336488:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11184.119151] LustreError: 336488:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11184.672046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11184.885442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11188.263365] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [11195.109230] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11195.579224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11200.553689] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [11205.951758] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11207.076831] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11212.273689] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11213.592680] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11219.946153] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [11219.964891] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11219.978894] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11219.990503] Lustre: Skipped 2 previous similar messages [11223.973565] Lustre: server umount lustre-OST0000 complete [11230.693729] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11230.711102] Lustre: Skipped 2 previous similar messages [11232.914410] Lustre: server umount lustre-MDT0000 complete [11235.808518] LustreError: 336487:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11235.827040] LustreError: 336487:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11235.899261] LustreError: 336467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788808705 with bad export cookie 3015845090717232744 [11235.907670] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11235.913147] LustreError: 336467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11236.158955] Lustre: server umount lustre-MDT0001 complete [11244.733516] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [11246.750226] Key type lgssc unregistered [11246.995626] LNet: 339921:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11246.998881] LNetError: 339921:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11248.038846] LNet: Removed LNI 192.168.201.152@tcp [11248.656624] Key type .llcrypt unregistered [11248.658886] Key type ._llcrypt unregistered [11263.384285] Key type ._llcrypt registered [11263.386562] Key type .llcrypt registered [11263.477353] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [11274.903901] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [11275.533697] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11275.626240] alg: No test for adler32 (adler32-zlib) [11276.667911] Lustre: Lustre: Build Version: 2.17.58_39_gc924a9e [11276.871194] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [11278.591220] Key type lgssc registered [11279.560768] Lustre: Echo OBD driver; http://www.lustre.org/ [11309.047656] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11309.086598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11310.294912] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11310.317712] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11310.375959] Lustre: lustre-MDT0000: new disk, initializing [11310.435764] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11310.449266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11314.504924] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [11323.354292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11323.463207] Lustre: 343059:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11323.501809] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11323.504455] Lustre: Skipped 1 previous similar message [11323.552620] Lustre: lustre-MDT0001: new disk, initializing [11323.627167] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11323.659468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11323.667268] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11327.409020] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [11334.873634] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11339.153842] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11344.528182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11344.758928] Lustre: lustre-OST0000: new disk, initializing [11344.762819] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11344.768356] Lustre: 345024:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11344.841700] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11349.796617] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [11353.136212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11353.143969] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11353.212511] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11357.226791] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11360.880556] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11361.018497] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11364.546832] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11364.662673] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11368.420022] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11368.429798] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11368.438947] Lustre: Skipped 1 previous similar message [11368.452606] Lustre: Skipped 1 previous similar message [11373.542622] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11373.557570] Lustre: Skipped 1 previous similar message [11378.659567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11378.664882] Lustre: Skipped 1 previous similar message [11380.191655] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11380.300823] Lustre: server umount lustre-OST0000 complete [11383.786703] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11383.789865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11383.802152] Lustre: Skipped 1 previous similar message [11383.817205] Lustre: Skipped 1 previous similar message [11388.897326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11388.903378] Lustre: Skipped 1 previous similar message [11389.783773] Lustre: server umount lustre-MDT0000 complete [11392.757476] LustreError: 343050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788808862 with bad export cookie 10817681636008907546 [11392.777038] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11393.024982] Lustre: server umount lustre-MDT0001 complete [11399.708154] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 15:21:08 (1788808868) [11412.126858] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [11419.201244] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [11445.917588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11446.108776] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11446.129523] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11446.194648] Lustre: lustre-MDT0000: new disk, initializing [11446.257595] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11446.277328] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11449.694344] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [11458.918402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11459.048420] Lustre: 349779:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11459.099540] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11459.103723] Lustre: Skipped 1 previous similar message [11459.179152] Lustre: lustre-MDT0001: new disk, initializing [11459.250582] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11459.287042] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11459.298573] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11462.930735] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [11470.803532] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11475.232154] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11480.458890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11480.770791] Lustre: lustre-OST0000: new disk, initializing [11480.774464] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11480.780436] Lustre: 351746:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11480.884476] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11485.784378] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [11488.814964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11488.830505] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11488.948477] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11493.359268] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11519.973075] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11519.979385] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11519.985182] LustreError: Skipped 1 previous similar message [11519.986054] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11520.002967] Lustre: Skipped 1 previous similar message [11520.026722] Lustre: Skipped 1 previous similar message [11522.874313] Lustre: server umount lustre-OST0000 complete [11525.992653] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [11526.002854] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11530.214655] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11532.297823] Lustre: server umount lustre-MDT0000 complete [11535.192229] LustreError: 349770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788809005 with bad export cookie 10817681636008908428 [11535.201452] LustreError: 349770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [11535.202161] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11535.564393] Lustre: server umount lustre-MDT0001 complete [11545.992719] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [11548.210143] Key type lgssc unregistered [11548.423034] LNet: 353899:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11548.431948] LNetError: 353899:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11549.479291] LNet: Removed LNI 192.168.201.152@tcp [11550.377273] Key type .llcrypt unregistered [11550.379240] Key type ._llcrypt unregistered [11624.336770] Key type ._llcrypt registered [11624.338305] Key type .llcrypt registered [11624.411180] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [11624.622863] Key type .llcrypt unregistered [11624.625297] Key type ._llcrypt unregistered [11684.407274] Key type ._llcrypt registered [11684.409457] Key type .llcrypt registered [11684.501417] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [11684.695266] Key type .llcrypt unregistered [11684.698654] Key type ._llcrypt unregistered [11708.996518] Key type ._llcrypt registered [11709.002103] Key type .llcrypt registered [11709.092871] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [11709.316347] Key type .llcrypt unregistered [11709.317866] Key type ._llcrypt unregistered [11736.810577] Key type ._llcrypt registered [11736.812978] Key type .llcrypt registered [11736.903373] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [11737.129352] Key type .llcrypt unregistered [11737.132602] Key type ._llcrypt unregistered [11800.152466] Key type ._llcrypt registered [11800.158370] Key type .llcrypt registered [11800.250952] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [11800.503466] Key type .llcrypt unregistered [11800.504680] Key type ._llcrypt unregistered [11824.019761] Key type ._llcrypt registered [11824.022100] Key type .llcrypt registered [11824.106887] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [11824.298379] Key type .llcrypt unregistered [11824.300112] Key type ._llcrypt unregistered [11848.616906] Key type ._llcrypt registered [11848.619402] Key type .llcrypt registered [11848.720135] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [11848.961324] Key type .llcrypt unregistered [11848.963090] Key type ._llcrypt unregistered [11872.408889] Key type ._llcrypt registered [11872.410903] Key type .llcrypt registered [11872.500661] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [11872.732502] Key type .llcrypt unregistered [11872.736108] Key type ._llcrypt unregistered [11896.427220] Key type ._llcrypt registered [11896.429833] Key type .llcrypt registered [11896.513192] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [11896.778611] Key type .llcrypt unregistered [11896.781820] Key type ._llcrypt unregistered [11921.033840] Key type ._llcrypt registered [11921.036536] Key type .llcrypt registered [11921.123425] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [11921.327518] Key type .llcrypt unregistered [11921.329336] Key type ._llcrypt unregistered [11967.546215] Key type ._llcrypt registered [11967.548431] Key type .llcrypt registered [11967.625925] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [11967.795377] Key type .llcrypt unregistered [11967.798093] Key type ._llcrypt unregistered [11989.055562] Key type ._llcrypt registered [11989.057737] Key type .llcrypt registered [11989.168218] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [11989.367341] Key type .llcrypt unregistered [11989.370707] Key type ._llcrypt unregistered [12011.416945] Key type ._llcrypt registered [12011.422570] Key type .llcrypt registered [12011.536221] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [12011.731593] Key type .llcrypt unregistered [12011.734183] Key type ._llcrypt unregistered [12049.533401] Key type ._llcrypt registered [12049.535468] Key type .llcrypt registered [12049.608589] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [12060.412287] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [12060.898500] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12060.911930] alg: No test for adler32 (adler32-zlib) [12061.853715] Lustre: Lustre: Build Version: 2.17.58_39_gc924a9e [12062.036706] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [12063.687213] Key type lgssc registered [12064.396379] Lustre: Echo OBD driver; http://www.lustre.org/ [12088.414698] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12088.438094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12089.606527] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [12089.632321] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [12089.682318] Lustre: tst250fs-MDT0000: new disk, initializing [12089.719991] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12089.740815] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12092.714183] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [12099.894131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12099.964363] Lustre: 373813:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12099.993797] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [12099.996553] Lustre: Skipped 1 previous similar message [12100.046391] Lustre: tst250fs-MDT0001: new disk, initializing [12100.096691] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12100.125258] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12100.141883] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12102.731054] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [12106.049440] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12110.404079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12110.663708] Lustre: tst250fs-OST0000: new disk, initializing [12110.668454] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [12110.674325] Lustre: 375446:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12110.764417] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12115.516844] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [12116.007289] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12116.017775] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12116.144775] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12123.008072] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12123.175905] Lustre: tst250fs-OST0001: new disk, initializing [12123.181369] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [12123.185965] Lustre: 376317:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12123.275216] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12126.173676] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12126.186756] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12126.249915] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12127.402781] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [12131.674213] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [12133.826551] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [12133.829087] Lustre: Skipped 2 previous similar messages [12151.267261] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12151.279466] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12151.289871] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12151.784897] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12151.794439] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12151.797818] Lustre: Skipped 1 previous similar message [12151.806654] Lustre: Skipped 2 previous similar messages [12153.314447] Lustre: server umount tst250fs-MDT0000 complete [12156.391564] LustreError: 373808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788809626 with bad export cookie 13632988233986586791 [12156.394243] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12156.407266] LustreError: 373808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12156.670576] Lustre: server umount tst250fs-MDT0001 complete [12169.552479] Lustre: server umount tst250fs-OST0000 complete [12182.387986] Lustre: server umount tst250fs-OST0001 complete [12194.128784] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [12199.981750] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [12221.669682] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 15:34:51 (1788809691) [12222.684624] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [12234.095950] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [12240.898597] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [12262.580514] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 11951 sec ========= 15:35:32 (1788809732) [12263.810429] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 15:35:33 (1788809733) === [12266.599672] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 15:35:36 (1788809736) === [12295.497520] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [12297.532908] Key type lgssc unregistered [12297.763426] LNet: 385444:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12297.770565] LNetError: 385444:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12298.792712] LNet: Removed LNI 192.168.201.152@tcp [12299.393199] Key type .llcrypt unregistered [12299.394555] Key type ._llcrypt unregistered