[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 461908370 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003179] x2apic enabled [ 0.004011] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.008777] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009025] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010014] pid_max: default: 32768 minimum: 301 [ 0.011140] LSM: Security Framework initializing [ 0.012055] Yama: becoming mindful. [ 0.013034] SELinux: Initializing. [ 0.014067] *** VALIDATE selinux *** [ 0.022767] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027191] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029046] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030107] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031107] *** VALIDATE tmpfs *** [ 0.032468] *** VALIDATE proc *** [ 0.034203] *** VALIDATE cgroup *** [ 0.035011] *** VALIDATE cgroup2 *** [ 0.036283] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037165] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039030] Spectre V2 : User space: Vulnerable [ 0.040007] Speculative Store Bypass: Vulnerable [ 0.043330] debug: unmapping init [mem 0xffffffffa8059000-0xffffffffa8060fff] [ 0.046125] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047700] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048030] ... version: 2 [ 0.049013] ... bit width: 48 [ 0.050011] ... generic registers: 4 [ 0.051012] ... value mask: 0000ffffffffffff [ 0.052012] ... max period: 00007fffffffffff [ 0.053015] ... fixed-purpose events: 3 [ 0.054012] ... event mask: 000000070000000f [ 0.055312] rcu: Hierarchical SRCU implementation. [ 0.057428] smp: Bringing up secondary CPUs ... [ 0.058580] x86: Booting SMP configuration: [ 0.059026] .... node #0, CPUs: #1 #2 #3 [ 0.062350] smp: Brought up 1 node, 4 CPUs [ 0.064013] smpboot: Max logical packages: 1 [ 0.065020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.150754] node 0 deferred pages initialised in 82ms [ 0.154108] devtmpfs: initialized [ 0.155289] x86/mm: Memory block size: 128MB [ 0.158741] gcov: version magic: 0x41383552 [ 0.161225] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.165120] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.167324] pinctrl core: initialized pinctrl subsystem [ 0.170201] [ 0.170822] ************************************************************* [ 0.173013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.176012] ** ** [ 0.178014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.180016] ** ** [ 0.182013] ** This means that this kernel is built to expose internal ** [ 0.184012] ** IOMMU data structures, which may compromise security on ** [ 0.186014] ** your system. ** [ 0.189017] ** ** [ 0.191012] ** If you see this message and you are not debugging the ** [ 0.193013] ** kernel, report this immediately to your vendor! ** [ 0.195014] ** ** [ 0.198013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.200012] ************************************************************* [ 0.203678] NET: Registered protocol family 16 [ 0.205439] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.208057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.211063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.215030] cpuidle: using governor menu [ 0.216563] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.217437] PCI: Using configuration type 1 for base access [ 0.218130] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.224218] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.227137] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.230076] cryptd: max_cpu_qlen set to 1000 [ 0.233159] ACPI: Added _OSI(Module Device) [ 0.234022] ACPI: Added _OSI(Processor Device) [ 0.236015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.237012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.242994] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.249315] ACPI: Interpreter enabled [ 0.251083] ACPI: PM: (supports S0 S3 S4 S5) [ 0.253034] ACPI: Using IOAPIC for interrupt routing [ 0.254117] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.257407] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.268401] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.270040] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.273017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.276084] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.282482] acpiphp: Slot [2] registered [ 0.283098] acpiphp: Slot [5] registered [ 0.285082] acpiphp: Slot [6] registered [ 0.286131] acpiphp: Slot [7] registered [ 0.287095] acpiphp: Slot [8] registered [ 0.288067] acpiphp: Slot [9] registered [ 0.289071] acpiphp: Slot [10] registered [ 0.290066] acpiphp: Slot [3] registered [ 0.292064] acpiphp: Slot [4] registered [ 0.293085] acpiphp: Slot [11] registered [ 0.294115] acpiphp: Slot [12] registered [ 0.295069] acpiphp: Slot [13] registered [ 0.296074] acpiphp: Slot [14] registered [ 0.297103] acpiphp: Slot [15] registered [ 0.298066] acpiphp: Slot [16] registered [ 0.299062] acpiphp: Slot [17] registered [ 0.301080] acpiphp: Slot [18] registered [ 0.302062] acpiphp: Slot [19] registered [ 0.303054] acpiphp: Slot [20] registered [ 0.304083] acpiphp: Slot [21] registered [ 0.304982] acpiphp: Slot [22] registered [ 0.306083] acpiphp: Slot [23] registered [ 0.307065] acpiphp: Slot [24] registered [ 0.308056] acpiphp: Slot [25] registered [ 0.309076] acpiphp: Slot [26] registered [ 0.310107] acpiphp: Slot [27] registered [ 0.311065] acpiphp: Slot [28] registered [ 0.311930] acpiphp: Slot [29] registered [ 0.313053] acpiphp: Slot [30] registered [ 0.313941] acpiphp: Slot [31] registered [ 0.315111] PCI host bridge to bus 0000:00 [ 0.317020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.319032] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.322026] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.325023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.327022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.330026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.332187] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.335026] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.339000] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.348807] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.354433] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.357019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.359032] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.362017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.365360] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.368957] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.372046] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.375845] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.381016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.396019] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.402998] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.408878] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.437022] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.450016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.494019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.527568] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.542017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.560027] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.593021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.609841] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.623017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.640022] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.663019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.678000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.684014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.690019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.711017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.720137] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.730018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.737015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.762018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.778109] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.795030] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.801016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.819018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.828941] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.831381] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.834415] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.836375] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.839257] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.844065] iommu: Default domain type: Passthrough [ 0.845457] SCSI subsystem initialized [ 0.846138] ACPI: bus type USB registered [ 0.848126] usbcore: registered new interface driver usbfs [ 0.850079] usbcore: registered new interface driver hub [ 0.851080] usbcore: registered new device driver usb [ 0.853172] pps_core: LinuxPPS API ver. 1 registered [ 0.855010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.858060] PTP clock support registered [ 0.860078] EDAC MC: Ver: 3.0.0 [ 0.862146] PCI: Using ACPI for IRQ routing [ 0.863897] NetLabel: Initializing [ 0.864012] NetLabel: domain hash size = 128 [ 0.865010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.866085] NetLabel: unlabeled traffic allowed by default [ 0.867132] vgaarb: loaded [ 0.869303] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.871017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.876402] clocksource: Switched to clocksource kvm-clock [ 0.979444] VFS: Disk quotas dquot_6.6.0 [ 0.980532] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.982196] *** VALIDATE ramfs *** [ 0.983058] *** VALIDATE hugetlbfs *** [ 0.984136] pnp: PnP ACPI init [ 0.986057] pnp: PnP ACPI: found 6 devices [ 1.002289] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.006015] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.008408] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.010935] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.013328] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.015687] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.017695] NET: Registered protocol family 2 [ 1.019504] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.023022] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.025499] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.029268] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.031724] TCP: Hash tables configured (established 65536 bind 65536) [ 1.033650] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.035794] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.037569] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.039532] NET: Registered protocol family 1 [ 1.041321] RPC: Registered named UNIX socket transport module. [ 1.043471] RPC: Registered udp transport module. [ 1.045184] RPC: Registered tcp transport module. [ 1.046989] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.049320] NET: Registered protocol family 44 [ 1.051080] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.053221] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.055223] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.057534] PCI: CLS 0 bytes, default 64 [ 1.058903] Unpacking initramfs... [ 2.382509] debug: unmapping init [mem 0xffff8909fcc54000-0xffff8909fffbffff] [ 2.386298] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.388376] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.391060] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.874426] Initialise system trusted keyrings [ 2.876150] Key type blacklist registered [ 2.878054] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.887223] zbud: loaded [ 2.890658] *** VALIDATE nfs *** [ 2.891992] *** VALIDATE nfs4 *** [ 2.893647] pstore: using deflate compression [ 2.898331] Platform Keyring initialized [ 2.992693] NET: Registered protocol family 38 [ 2.994576] Key type asymmetric registered [ 2.996222] Asymmetric key parser 'x509' registered [ 2.997991] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.000687] io scheduler mq-deadline registered [ 3.001622] io scheduler kyber registered [ 3.002594] io scheduler bfq registered [ 3.004476] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.007463] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.010208] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.012805] ACPI: Power Button [PWRF] [ 3.018224] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.024887] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.036868] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.042809] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.056300] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.086631] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.115079] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.124166] Non-volatile memory driver v1.3 [ 3.125690] Linux agpgart interface v0.103 [ 3.161385] virtio_blk virtio1: [vda] 146168 512-byte logical blocks (74.8 MB/71.4 MiB) [ 3.164544] vda: detected capacity change from 0 to 74838016 [ 3.179661] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.182112] vdb: detected capacity change from 0 to 1073741824 [ 3.194961] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.197456] vdc: detected capacity change from 0 to 2621440000 [ 3.210649] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.213654] vdd: detected capacity change from 0 to 2621440000 [ 3.228368] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.231480] vde: detected capacity change from 0 to 4294967296 [ 3.247320] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.250326] vdf: detected capacity change from 0 to 4294967296 [ 3.261564] libphy: Fixed MDIO Bus: probed [ 3.269728] usbcore: registered new interface driver usbserial_generic [ 3.272578] usbserial: USB Serial support registered for generic [ 3.275493] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.279873] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.281790] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.284264] mousedev: PS/2 mouse device common for all mice [ 3.287235] rtc_cmos 00:05: RTC can wake from S4 [ 3.290063] rtc_cmos 00:05: registered as rtc0 [ 3.292393] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.293432] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.297831] intel_pstate: CPU model not supported [ 3.300902] hid: raw HID events driver (C) Jiri Kosina [ 3.308533] usbcore: registered new interface driver usbhid [ 3.310209] usbhid: USB HID core driver [ 3.312290] drop_monitor: Initializing network drop monitor service [ 3.314689] Initializing XFRM netlink socket [ 3.316607] NET: Registered protocol family 10 [ 3.320045] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.322304] Segment Routing with IPv6 [ 3.324458] NET: Registered protocol family 17 [ 3.327780] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.331884] mpls_gso: MPLS GSO support [ 3.336730] RAS: Correctable Errors collector initialized. [ 3.338804] AVX version of gcm_enc/dec engaged. [ 3.340402] AES CTR mode by8 optimization enabled [ 3.418145] sched_clock: Marking stable (3418113937, 0)->(4322838625, -904724688) [ 3.422236] registered taskstats version 1 [ 3.424309] Loading compiled-in X.509 certificates [ 3.426232] zswap: loaded using pool lzo/zbud [ 3.449193] Key type big_key registered [ 3.460063] Key type encrypted registered [ 3.461146] ima: No TPM chip found, activating TPM-bypass! [ 3.462505] ima: Allocated hash algorithm: sha1 [ 3.463597] ima: No architecture policies found [ 3.465083] evm: Initialising EVM extended attributes: [ 3.466320] evm: security.selinux [ 3.467114] evm: security.ima [ 3.468195] evm: security.capability [ 3.469538] evm: HMAC attrs: 0x1 [ 3.471969] rtc_cmos 00:05: setting system clock to 2026-09-15 12:59:32 UTC (1789477172) [ 3.477923] debug: unmapping init [mem 0xffffffffa9003000-0xffffffffa91fffff] [ 3.480961] debug: unmapping init [mem 0xffffffffa7d82000-0xffffffffa8058fff] [ 3.490088] Write protecting the kernel read-only data: 28672k [ 3.493494] debug: unmapping init [mem 0xffffffffa6403000-0xffffffffa65fffff] [ 3.496131] debug: unmapping init [mem 0xffffffffa6d14000-0xffffffffa6dfffff] [ 3.529457] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.537028] systemd[1]: Detected virtualization kvm. [ 3.538585] systemd[1]: Detected architecture x86-64. [ 3.539854] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.570431] systemd[1]: No hostname configured. [ 3.572071] systemd[1]: Set hostname to . [ 3.574467] random: systemd: uninitialized urandom read (16 bytes read) [ 3.576703] systemd[1]: Initializing machine ID from random generator. [ 3.617146] random: ln: uninitialized urandom read (6 bytes read) [ 3.699903] random: systemd: uninitialized urandom read (16 bytes read) [ 3.702709] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.706417] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.709737] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... Starting Journal Service... [ OK ] Reached target Timers. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.240586] device-mapper: uevent: version 1.0.3 [ 4.242786] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.900966] virtio_net virtio0 ens2: renamed from eth0 [ 4.927510] random: fast init done [ 4.966449] scsi host0: ata_piix [ 4.988200] scsi host1: ata_piix [ 4.989584] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.992049] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.551928] dracut-initqueue[587]: RTNETLINK answers: File exists [ 9.978924] random: crng init done [ 9.980265] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.247037] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.255153] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.501810] SELinux: Disabled at runtime. [ 11.561509] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.568804] systemd[1]: Detected virtualization kvm. [ 11.570761] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.007548] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.010907] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.015479] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.019544] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.023173] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.031237] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.043392] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Remount Root and Kernel File Systems... Mounting POSIX Message Queue File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Huge Pages File System... [ 12.113795] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Kernel Debug File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Created slice system-getty.slice. [ OK ] Reached target Slices. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.449743] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.694518] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.706846] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.830572] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.848715] EDAC sbridge: Ver: 1.1.2 [ 14.448625] Key type dns_resolver registered [ 14.758747] NFS: Registering the id_resolver key type [ 14.760676] Key type id_resolver registered [ 14.761973] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg233-server login: [ 72.454914] libcfs: loading out-of-tree module taints kernel. [ 72.503636] Key type ._llcrypt registered [ 72.505638] Key type .llcrypt registered [ 72.659828] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 94.085621] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 96.090664] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 96.106442] alg: No test for adler32 (adler32-zlib) [ 97.801394] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [ 99.078601] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 100.944360] Key type lgssc registered [ 103.141311] Lustre: Echo OBD driver; http://www.lustre.org/ [ 121.938859] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 134.572104] hrtimer: interrupt took 3638519 ns [ 169.922851] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 183.104674] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 183.148756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 184.289575] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 184.315888] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 184.377952] Lustre: lustre-MDT0000: new disk, initializing [ 184.428592] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 184.440271] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 189.349903] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 207.209469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 207.314898] Lustre: 6534:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 207.339525] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 207.344248] Lustre: Skipped 1 previous similar message [ 207.405946] Lustre: lustre-MDT0001: new disk, initializing [ 207.464262] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 207.489973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 207.500268] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 212.533856] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 217.545231] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 228.758509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 229.020146] Lustre: lustre-OST0000: new disk, initializing [ 229.026140] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 229.030317] Lustre: 8472:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 229.083040] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 230.953835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 230.973453] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 231.071932] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 235.758694] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 253.835167] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 253.957413] Lustre: lustre-OST0001: new disk, initializing [ 253.965435] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 253.971683] Lustre: 9547:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 254.055391] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 261.488622] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 261.726918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 261.750755] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 261.851248] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 274.057129] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 284.865769] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 293.599344] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing check_logdir /tmp/testlogs/ [ 299.933611] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing yml_node [ 304.026610] Lustre: DEBUG MARKER: Client: 2.17.57.84 [ 307.030142] Lustre: DEBUG MARKER: MDS: 2.17.57.84 [ 309.934106] Lustre: DEBUG MARKER: OSS: 2.17.57.84 [ 311.501024] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 15 09:04:38 EDT 2026 [ 332.001059] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 333.886921] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 350.692610] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 350.697450] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 350.726692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 354.276031] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 354.277414] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 354.288876] Lustre: Skipped 1 previous similar message [ 354.308131] Lustre: Skipped 1 previous similar message [ 356.700773] Lustre: server umount lustre-MDT0000 complete [ 361.477985] LustreError: 6527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789477530 with bad export cookie 7716483351973787683 [ 361.491106] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 361.492457] LustreError: 6527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 364.513283] LustreError: 6541:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 364.514959] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 364.516882] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 364.516890] Lustre: Skipped 2 previous similar messages [ 364.538478] LustreError: 6541:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 364.602647] Lustre: Skipped 2 previous similar messages [ 366.561789] LustreError: 6542:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 366.563607] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 366.584495] LustreError: 6542:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 366.603587] Lustre: Skipped 2 previous similar messages [ 368.198343] Lustre: server umount lustre-MDT0001 complete [ 373.845437] Lustre: server umount lustre-OST0000 complete [ 378.669993] Lustre: server umount lustre-OST0001 complete [ 386.226716] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 397.570699] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 440.527644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 440.783731] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 440.877355] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 441.199601] Lustre: lustre-MDT0000: new disk, initializing [ 441.402354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 441.439874] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 446.994956] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 459.614186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 459.764197] Lustre: 15101:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 459.910450] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 459.913219] Lustre: Skipped 1 previous similar message [ 460.087253] Lustre: lustre-MDT0001: new disk, initializing [ 460.263637] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 460.342840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 460.358463] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 465.699113] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 477.385252] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 483.399356] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 491.650626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 492.142822] Lustre: lustre-OST0000: new disk, initializing [ 492.146964] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 492.165768] Lustre: 17068:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 492.286275] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 493.437214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 493.455335] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 493.660083] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 500.483492] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 511.287568] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 516.375272] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 516.617309] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 521.735400] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 521.930198] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 524.265271] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 524.270650] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 524.288344] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 524.302558] Lustre: Skipped 1 previous similar message [ 534.499692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 534.510163] Lustre: Skipped 3 previous similar messages [ 538.080166] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 538.512900] Lustre: server umount lustre-OST0000 complete [ 544.759287] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 544.781208] Lustre: Skipped 2 previous similar messages [ 549.506300] Lustre: server umount lustre-MDT0000 complete [ 549.859683] LustreError: 15113:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 549.881947] LustreError: 15113:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 554.150071] LustreError: 15095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789477723 with bad export cookie 7716483351973789216 [ 554.166336] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 554.174739] LustreError: 15095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 554.680822] Lustre: server umount lustre-MDT0001 complete [ 561.803904] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 09:08:48 (1789477728) [ 583.915945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 584.384109] LustreError: 19946:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 584.470639] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 589.801472] LustreError: 19947:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 590.612457] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 600.035955] LustreError: 19946:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 600.058363] LustreError: 19946:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 601.567495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 607.540552] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 617.772608] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 625.282455] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 628.635599] Lustre: Failing over lustre-MDT0000 [ 629.352637] Lustre: server umount lustre-MDT0000 complete [ 631.777464] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 631.791464] LustreError: 19942:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 631.811085] LustreError: 19942:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 647.136111] Lustre: 3668:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789477800/real 1789477800] req@ffff890a774b8700 x1876402915940608/t0(0) o400->MGC192.168.202.133@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789477816 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 647.185676] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 649.954807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 657.752186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 657.758593] Lustre: Skipped 1 previous similar message [ 657.855888] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 663.024621] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 663.031671] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 663.078983] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 664.054670] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 675.528721] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 681.441788] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 681.447319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 681.467429] Lustre: Skipped 2 previous similar messages [ 681.486920] Lustre: Skipped 3 previous similar messages [ 686.762683] Lustre: server umount lustre-MDT0000 complete [ 691.468251] LustreError: 19927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789477860 with bad export cookie 7716483351973790630 [ 691.501288] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 691.505233] LustreError: 19927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 691.866816] Lustre: server umount lustre-MDT0001 complete [ 702.987739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 703.562366] LustreError: 23290:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 703.575475] LustreError: 23290:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 709.282130] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 720.984922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 725.342684] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 734.463793] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 740.695925] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 743.451088] Lustre: Failing over lustre-MDT0000 [ 743.698797] Lustre: server umount lustre-MDT0000 complete [ 746.530117] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 762.722805] Lustre: 3667:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789477915/real 1789477915] req@ffff890942b65500 x1876402915971072/t0(0) o400->MGC192.168.202.133@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789477931 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 762.769325] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 764.600498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 767.906198] LustreError: 23285:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 767.923520] LustreError: 23285:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 16 previous similar messages [ 772.379316] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 777.708492] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 777.720059] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 777.735962] Lustre: Skipped 1 previous similar message [ 777.781739] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 778.216607] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 791.468945] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 798.177364] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 798.181415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 798.184275] Lustre: Skipped 2 previous similar messages [ 802.794478] Lustre: server umount lustre-MDT0000 complete [ 807.071289] LustreError: 23272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789477976 with bad export cookie 7716483351973791568 [ 807.097872] LustreError: 23272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 807.115804] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 807.728944] Lustre: server umount lustre-MDT0001 complete [ 818.456332] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 09:13:05 (1789477985) [ 839.835756] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 854.834347] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 899.616722] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 910.882984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 911.205528] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 911.283987] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 911.416438] Lustre: lustre-MDT0000: new disk, initializing [ 911.553146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 911.562344] Lustre: Skipped 3 previous similar messages [ 911.617944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 916.287255] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 928.764854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 928.925698] Lustre: 29682:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 929.412589] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 935.508075] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 947.451196] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 953.972556] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 961.666778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 962.054174] Lustre: 31651:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 963.755089] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 963.815127] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 963.936759] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 963.954792] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 969.028700] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 979.545264] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 985.169992] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 1010.147797] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1010.159303] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1010.168629] LustreError: Skipped 1 previous similar message [ 1010.183937] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1010.190631] Lustre: Skipped 4 previous similar messages [ 1010.221928] Lustre: Skipped 3 previous similar messages [ 1020.898128] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1021.107449] Lustre: server umount lustre-OST0000 complete [ 1026.531793] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1032.134988] Lustre: server umount lustre-MDT0000 complete [ 1035.748146] LustreError: 29688:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1035.785269] LustreError: 29688:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 1036.717811] LustreError: 29674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789478205 with bad export cookie 7716483351973791974 [ 1036.727728] LustreError: 29674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1036.732782] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1037.556184] Lustre: server umount lustre-MDT0001 complete [ 1051.133517] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 1054.529915] Key type lgssc unregistered [ 1054.788468] LNet: 33638:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1054.802554] LNetError: 33638:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1055.846758] LNet: Removed LNI 192.168.202.133@tcp [ 1056.700175] Key type .llcrypt unregistered [ 1056.702462] Key type ._llcrypt unregistered [ 1085.214122] Key type ._llcrypt registered [ 1085.215549] Key type .llcrypt registered [ 1085.365876] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 1086.487432] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1086.511135] alg: No test for adler32 (adler32-zlib) [ 1087.628578] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [ 1088.007191] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 1089.705196] Key type lgssc registered [ 1090.660707] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1099.806198] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1099.827110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1101.648704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1107.885707] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1111.012167] LustreError: 35055:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1111.030982] LustreError: 35055:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1116.130808] LustreError: 35056:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1116.996723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1117.508322] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1122.847794] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1131.656877] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1137.862451] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1145.743181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1146.247791] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1154.310652] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1163.611155] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1163.680392] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1180.716418] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 09:19:07 (1789478347) [ 1182.753192] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1184.945402] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1192.655498] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 09:19:19 (1789478359) [ 1201.876651] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 09:19:29 (1789478369) [ 1203.069237] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1212.517569] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1212.531060] Lustre: Skipped 2 previous similar messages [ 1215.219851] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 09:19:41 (1789478381) [ 1218.036361] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1218.039403] Lustre: Skipped 2 previous similar messages [ 1227.844130] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1227.856885] Lustre: Skipped 4 previous similar messages [ 1246.096641] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 09:20:12 (1789478412) [ 1249.263579] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1253.593585] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1264.155818] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 09:20:31 (1789478431) [ 1267.001863] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1267.013360] Lustre: Skipped 6 previous similar messages [ 1283.896276] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 09:20:50 (1789478450) [ 1300.966097] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1300.978955] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1300.998653] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1301.992659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1301.994703] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1302.002276] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1302.006699] Lustre: Skipped 1 previous similar message [ 1304.303433] Lustre: server umount lustre-OST0000 complete [ 1312.227169] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1312.238757] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1312.242507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1312.252858] Lustre: Skipped 1 previous similar message [ 1312.284417] Lustre: Skipped 1 previous similar message [ 1315.662606] Lustre: server umount lustre-MDT0000 complete [ 1317.347473] LustreError: 35052:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1317.391330] LustreError: 35052:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1320.318561] LustreError: 38969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789478489 with bad export cookie 6676360430508063234 [ 1320.320192] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1320.331018] LustreError: 38969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1320.798766] Lustre: server umount lustre-MDT0001 complete [ 1335.599100] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 1339.302548] Key type lgssc unregistered [ 1339.698750] LNet: 42916:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1339.702409] LNetError: 42916:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1340.781291] LNet: Removed LNI 192.168.202.133@tcp [ 1342.082133] Key type .llcrypt unregistered [ 1342.084409] Key type ._llcrypt unregistered [ 1363.938447] Key type ._llcrypt registered [ 1363.940134] Key type .llcrypt registered [ 1364.028728] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 1383.408714] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 1384.369509] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1384.516669] alg: No test for adler32 (adler32-zlib) [ 1385.805771] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [ 1386.187068] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 1387.968600] Key type lgssc registered [ 1389.334150] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1433.536070] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1433.569163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1434.973947] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1435.030313] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1435.129869] Lustre: lustre-MDT0000: new disk, initializing [ 1435.227134] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1435.244438] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1440.665811] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1453.331814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1453.503582] Lustre: 46080:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1453.560485] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1453.563769] Lustre: Skipped 1 previous similar message [ 1453.668860] Lustre: lustre-MDT0001: new disk, initializing [ 1453.820414] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1453.906601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1453.918413] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1459.546818] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1471.130864] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1477.884988] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1486.091861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1486.486163] Lustre: lustre-OST0000: new disk, initializing [ 1486.490923] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1486.497440] Lustre: 48049:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1486.628932] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1492.072888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1492.084728] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1492.225893] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1493.325495] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1504.456561] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1511.346574] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1511.801783] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1518.265845] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1518.632949] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1523.172747] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1523.174078] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1523.184511] LustreError: Skipped 1 previous similar message [ 1523.191740] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1523.220883] Lustre: Skipped 1 previous similar message [ 1527.643626] Lustre: server umount lustre-OST0000 complete [ 1533.410478] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1533.430543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1533.440958] Lustre: Skipped 2 previous similar messages [ 1538.529891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1539.768468] Lustre: server umount lustre-MDT0000 complete [ 1544.101616] LustreError: 46073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789478713 with bad export cookie 15391948093787054852 [ 1544.105164] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1544.112539] LustreError: 46073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1544.635693] Lustre: server umount lustre-MDT0001 complete [ 1556.614426] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 09:25:23 (1789478723) [ 1564.467944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1565.097500] LustreError: 50509:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1565.129766] LustreError: 50509:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1565.200097] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1570.276409] LustreError: 50510:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1570.616792] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1575.397268] LustreError: 50509:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1579.707032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1580.187387] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1584.786964] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1592.039690] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1598.494303] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1605.042713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1605.282064] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1611.263465] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1621.129223] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1629.756923] LustreError: 53036:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1630.926945] Lustre: Modifying parameter general.timeout=1 in log params [ 1632.312166] Lustre: Modifying parameter general.timeout=2 in log params [ 1635.338909] Lustre: Modifying parameter general.timeout=5 in log params [ 1635.341795] Lustre: Skipped 2 previous similar messages [ 1640.139297] Lustre: Modifying parameter general.timeout=10 in log params [ 1640.145812] Lustre: Skipped 4 previous similar messages [ 1648.870831] Lustre: Modifying parameter general.timeout=18 in log params [ 1648.876286] Lustre: Skipped 7 previous similar messages [ 1665.052598] Lustre: Modifying parameter general.timeout=35 in log params [ 1665.054488] Lustre: Skipped 16 previous similar messages [ 1697.152751] Lustre: Modifying parameter general.timeout=62 in log params [ 1697.162435] Lustre: Skipped 26 previous similar messages [ 1761.616910] Lustre: Modifying parameter general.timeout=125 in log params [ 1761.627175] Lustre: Skipped 62 previous similar messages [ 1778.399027] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 09:29:05 (1789478945) [ 1781.702267] Lustre: 59998:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1783.880723] Lustre: 60097:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1786.052937] Lustre: 60195:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1790.241388] Lustre: 60408:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1790.252560] Lustre: 60408:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1823.169698] Lustre: 61870:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1823.180346] Lustre: 61870:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1823.210076] LustreError: 61870:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1828.375873] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 09:29:55 (1789478995) [ 1846.242078] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1846.246466] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1846.249069] Lustre: Skipped 1 previous similar message [ 1846.251583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1846.251593] Lustre: Skipped 1 previous similar message [ 1851.363881] LustreError: 50509:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1851.396656] LustreError: 50509:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1851.543823] Lustre: server umount lustre-MDT0000 complete [ 1855.884303] LustreError: 50489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789479024 with bad export cookie 15391948093787055734 [ 1855.888540] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1855.893036] LustreError: 50489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1856.132386] Lustre: server umount lustre-MDT0001 complete [ 1870.273611] Lustre: server umount lustre-OST0000 complete [ 1897.421187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1897.752648] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1897.797590] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1897.808179] Lustre: Skipped 27 previous similar messages [ 1898.047288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1904.179697] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1914.231088] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1914.373909] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1914.460263] Lustre: 64404:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1919.508993] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1928.362169] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1935.041554] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1943.624878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1943.887809] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1944.284444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1944.298183] Lustre: Skipped 1 previous similar message [ 1951.360441] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1960.280936] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1962.411150] Lustre: 66971:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1971.681410] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1971.692768] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1971.727158] Lustre: Skipped 2 previous similar messages [ 1971.734100] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1971.742148] Lustre: Skipped 2 previous similar messages [ 1976.406789] Lustre: server umount lustre-OST0000 complete [ 1981.410697] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1981.420453] LustreError: Skipped 1 previous similar message [ 1981.425419] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1981.462728] Lustre: Skipped 1 previous similar message [ 1981.474796] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1981.481697] Lustre: Skipped 2 previous similar messages [ 1987.069327] Lustre: server umount lustre-MDT0000 complete [ 1987.553879] LustreError: 64415:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1987.593529] LustreError: 64415:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1991.047410] LustreError: 64395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789479160 with bad export cookie 15391948093787058940 [ 1991.054517] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1991.057907] LustreError: 64395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1991.409327] Lustre: server umount lustre-MDT0001 complete [ 2003.294510] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 2006.915475] Key type lgssc unregistered [ 2007.423841] LNet: 68186:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2007.455544] LNetError: 68186:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2008.489529] LNet: Removed LNI 192.168.202.133@tcp [ 2009.421409] Key type .llcrypt unregistered [ 2009.423202] Key type ._llcrypt unregistered [ 2035.593473] Key type ._llcrypt registered [ 2035.597334] Key type .llcrypt registered [ 2035.716248] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 2037.994461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2038.088173] alg: No test for adler32 (adler32-zlib) [ 2039.107930] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [ 2039.393689] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 2041.072759] Key type lgssc registered [ 2042.635075] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2052.065089] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2052.079593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2053.455075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2057.810727] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2063.848283] LustreError: 69601:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2063.871768] LustreError: 69601:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2066.679055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2067.290428] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2071.776729] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2080.118902] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2088.362273] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2100.747516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2102.581099] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2109.911948] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2121.455574] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2135.274500] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2148.835084] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2148.838248] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2148.847057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2148.866135] Lustre: Skipped 2 previous similar messages [ 2151.153233] Lustre: server umount lustre-MDT0000 complete [ 2153.960519] LustreError: 69601:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2153.973774] LustreError: 69601:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2156.260072] LustreError: 69580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789479325 with bad export cookie 12744487554888710210 [ 2156.274773] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2156.288675] LustreError: 69580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2156.552333] Lustre: server umount lustre-MDT0001 complete [ 2170.973544] Lustre: server umount lustre-OST0000 complete [ 2202.671178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2203.013773] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2203.044466] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2203.359320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2209.053843] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2220.040448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2220.177499] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2220.230182] Lustre: 74475:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2220.666131] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2227.074965] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2237.840359] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2244.818516] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2253.331867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2253.477437] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2253.810683] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2260.781655] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2268.803718] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2270.680338] Lustre: 77051:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2273.424818] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2282.976905] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2282.989286] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2283.011966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2283.018812] Lustre: Skipped 2 previous similar messages [ 2286.535290] Lustre: server umount lustre-OST0000 complete [ 2292.194732] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2292.203827] LustreError: Skipped 1 previous similar message [ 2292.208646] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2292.224834] Lustre: Skipped 1 previous similar message [ 2292.232168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2292.235051] Lustre: Skipped 2 previous similar messages [ 2297.464602] Lustre: server umount lustre-MDT0000 complete [ 2298.852050] LustreError: 74485:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2298.867993] LustreError: 74485:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2302.065746] LustreError: 74465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789479471 with bad export cookie 12744487554888711050 [ 2302.070221] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2302.088882] LustreError: 74465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2302.556046] Lustre: server umount lustre-MDT0001 complete [ 2313.943251] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 2317.476343] Key type lgssc unregistered [ 2317.875950] LNet: 78267:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2317.882885] LNetError: 78267:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2317.902226] LNet: Removed LNI 192.168.202.133@tcp [ 2319.231123] Key type .llcrypt unregistered [ 2319.233935] Key type ._llcrypt unregistered [ 2343.314099] Key type ._llcrypt registered [ 2343.316792] Key type .llcrypt registered [ 2343.471840] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 2344.200407] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2344.231398] alg: No test for adler32 (adler32-zlib) [ 2345.307758] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [ 2345.556462] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 2347.289307] Key type lgssc registered [ 2348.367706] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2356.648506] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2356.702897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2358.488816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2363.870107] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2368.995398] LustreError: 79681:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2369.024149] LustreError: 79681:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2373.060052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2373.592746] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2376.297688] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2384.078369] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2391.413285] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2399.389406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2399.897632] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2407.800446] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2416.243843] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2425.317592] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2426.105933] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2427.972567] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2427.990024] Lustre: Skipped 1 previous similar message [ 2430.267591] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2430.278737] Lustre: Skipped 1 previous similar message [ 2434.865115] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2434.867266] Lustre: Skipped 3 previous similar messages [ 2442.915714] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2442.927571] Lustre: Skipped 7 previous similar messages [ 2459.119637] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2459.129533] Lustre: Skipped 16 previous similar messages [ 2491.169197] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2491.183863] Lustre: Skipped 30 previous similar messages [ 2535.420892] Lustre: 87785:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2538.537434] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 09:41:45 (1789479705) [ 2540.361603] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2542.801292] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 09:41:49 (1789479709) [ 2547.681408] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2547.703539] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2547.729838] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2551.784407] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2551.784950] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2553.697594] Lustre: server umount lustre-OST0000 complete [ 2562.039660] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2562.050291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2562.060170] Lustre: Skipped 1 previous similar message [ 2562.088507] Lustre: Skipped 2 previous similar messages [ 2564.370298] Lustre: server umount lustre-MDT0000 complete [ 2567.156247] LustreError: 79682:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2567.183332] LustreError: 79682:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2568.936763] LustreError: 79662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789479737 with bad export cookie 5043634867890928763 [ 2568.939929] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2568.954518] LustreError: 79662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2569.446196] Lustre: server umount lustre-MDT0001 complete [ 2583.419675] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 2586.271628] Key type lgssc unregistered [ 2586.734586] LNet: 89037:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2586.743587] LNetError: 89037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2586.779389] LNet: Removed LNI 192.168.202.133@tcp [ 2587.905942] Key type .llcrypt unregistered [ 2587.907199] Key type ._llcrypt unregistered [ 2594.782285] Key type ._llcrypt registered [ 2594.787081] Key type .llcrypt registered [ 2595.005178] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2601.562842] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules [ 2603.020184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2603.152093] alg: No test for adler32 (adler32-zlib) [ 2604.484141] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [ 2605.077790] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 2606.875706] Key type lgssc registered [ 2608.506864] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2609.558318] LustreError: 90290:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2627.560557] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 2632.705420] LustreError: 90290:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2637.710282] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2637.766631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2639.623175] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2644.544153] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2649.122730] LustreError: 91314:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2649.144768] LustreError: 91314:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2654.191211] LustreError: 91315:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2654.792744] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 09:43:41 (1789479821) [ 2659.297985] LustreError: 91314:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2660.949133] LustreError: 91346:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2660.966651] LustreError: 91346:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2660.976750] LustreError: 91346:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 22, retries 0, failed: rc = -5 [ 2661.650458] Lustre: server umount lustre-MDT0000 complete [ 2680.671191] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 2684.323416] Key type lgssc unregistered [ 2684.641090] LNet: 92846:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2684.646430] LNetError: 92846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2684.664626] LNet: Removed LNI 192.168.202.133@tcp [ 2685.669673] Key type .llcrypt unregistered [ 2685.674580] Key type ._llcrypt unregistered [ 2704.071797] Key type ._llcrypt registered [ 2704.073929] Key type .llcrypt registered [ 2704.330483] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 2705.204410] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2705.226897] alg: No test for adler32 (adler32-zlib) [ 2706.433296] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [ 2706.656355] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 2708.416225] Key type lgssc registered [ 2710.099972] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2719.999864] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2720.042266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2721.730869] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2727.206896] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2730.978431] LustreError: 94066:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2731.010422] LustreError: 94066:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2736.098642] LustreError: 94065:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2736.800580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2737.526295] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2741.503974] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2748.498925] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2756.432648] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2765.536954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2766.850860] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2774.790887] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2785.818801] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2845.126604] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 09:46:51 (1789480011) [ 2847.530299] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2849.578848] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 09:46:56 (1789480016) [ 2854.881756] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2854.887942] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2854.908960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2858.984149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2858.986519] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2860.542699] Lustre: server umount lustre-MDT0000 complete [ 2864.100423] LustreError: 94066:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2864.145250] LustreError: 94066:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2864.764949] LustreError: 94046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789480033 with bad export cookie 13937112777164058853 [ 2864.783376] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2865.261603] Lustre: server umount lustre-MDT0001 complete [ 2879.876472] Lustre: server umount lustre-OST0000 complete [ 2893.945151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2894.724395] LustreError: 98373:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2894.956239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2899.939626] LustreError: 98374:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2901.479993] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2909.156063] LustreError: 98373:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2909.204375] LustreError: 98373:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2913.381492] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2914.557500] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2920.607861] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2929.003495] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2935.936611] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2952.700476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2953.044508] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2953.062650] LustreError: 98363:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2953.127248] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2953.152575] LustreError: 100458:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2953.162419] LustreError: 100458:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2953.173445] LustreError: 100458:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2953.257692] Lustre: server umount lustre-OST0000 complete [ 2953.265817] LustreError: 100458:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2963.959892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2964.206740] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2964.226160] LustreError: 98363:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2964.286310] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2964.294800] LustreError: 100924:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2964.303619] LustreError: 100924:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2964.310256] LustreError: 100924:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2964.406685] Lustre: server umount lustre-OST0000 complete [ 2964.411135] LustreError: 100924:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2975.580144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2975.845950] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2975.853153] Lustre: Found index 0 for lustre-OST0000, updating log [ 2975.855988] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2975.944945] Lustre: lustre-OST0000: new disk, initializing [ 2975.951148] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2975.959893] Lustre: 101458:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2976.364580] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2984.707912] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2989.252572] Lustre: Failing over lustre-OST0000 [ 2989.423344] Lustre: server umount lustre-OST0000 complete [ 2995.692229] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2995.692750] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2995.699853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2995.699864] Lustre: Skipped 2 previous similar messages [ 2995.711996] Lustre: Skipped 2 previous similar messages [ 3000.659988] Lustre: server umount lustre-MDT0000 complete [ 3000.814191] LustreError: 101468:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3000.849722] LustreError: 101468:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3006.077099] LustreError: 98353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789480175 with bad export cookie 13937112777164061597 [ 3006.094477] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3006.097519] LustreError: 98353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3017.007296] Lustre: server umount lustre-MDT0001 complete [ 3027.418185] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 09:49:54 (1789480194) [ 3048.472989] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 3062.526038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3063.422115] LustreError: 103812:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3063.439538] LustreError: 103812:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3063.690990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3071.130469] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3084.938636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3091.238577] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3094.603753] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 3103.655374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3105.494492] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3105.511255] Lustre: Skipped 1 previous similar message [ 3105.530352] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3108.603659] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3108.615564] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0000-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 3108.647239] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -16 [ 3110.700791] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0000-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:57 [ 3110.740485] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -16 [ 3110.766881] Lustre: lustre-OST0000: Recovery over after 0:02, of 1 clients 1 recovered and 0 were evicted. [ 3110.770460] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3113.891527] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3124.270457] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3124.405304] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3124.524331] Lustre: lustre-OST0001: new disk, initializing [ 3124.527653] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3124.537199] Lustre: 106132:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3132.114591] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3132.565582] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3132.579922] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3132.776565] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3140.876703] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3149.636946] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3155.430221] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 3155.439894] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3155.449615] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3155.451759] Lustre: Skipped 1 previous similar message [ 3160.616607] Lustre: server umount lustre-MDT0001 complete [ 3161.042166] LustreError: 103807:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3161.063562] LustreError: 103807:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3172.615537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3172.705611] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3172.710977] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3172.781543] Lustre: 103800:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3174.016495] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3174.027565] Lustre: Skipped 1 previous similar message [ 3174.295388] LustreError: 107654:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3174.311152] LustreError: 107654:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3174.345106] LustreError: 103804:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3174.594464] LustreError: 107670:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.133@tcp: cfg command failed: rc = -17 [ 3174.617972] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3174.617972] [ 3179.517525] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3180.489440] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3192.666539] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 09:52:39 (1789480359) [ 3214.906657] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 3236.109691] Lustre: 109752:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3264.274836] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3268.547302] Lustre: 110887:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3357.665833] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3357.677689] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3357.700371] Lustre: Skipped 2 previous similar messages [ 3357.713520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3357.720720] Lustre: Skipped 4 previous similar messages [ 3364.053268] Lustre: server umount lustre-MDT0000 complete [ 3368.404832] LustreError: 103794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789480537 with bad export cookie 13937112777164062598 [ 3368.406912] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3368.420535] LustreError: 103794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3368.897589] Lustre: server umount lustre-MDT0001 complete [ 3384.301055] Lustre: 93378:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789480537/real 1789480537] req@ffff89094cf28a80 x1876405649792128/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789480553 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3384.336294] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3384.349846] Lustre: Skipped 3 previous similar messages [ 3384.467553] Lustre: server umount lustre-OST0000 complete [ 3389.078507] Lustre: server umount lustre-OST0001 complete [ 3395.689902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3406.862262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3416.599837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3427.791874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3449.117875] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 3461.127472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3461.165115] Lustre: lustre-MDT0000: reset Object Index mappings [ 3461.863584] LustreError: 114843:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3461.881940] LustreError: 114843:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 3462.080821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3467.773666] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3478.587296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3478.632080] Lustre: lustre-MDT0001: reset Object Index mappings [ 3485.244135] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3489.555383] Lustre: 115991:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3497.740611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3504.889383] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:232 to 0x280000400:257) [ 3506.943807] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:349 to 0x280000401:386) [ 3507.360432] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3517.963944] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3524.081377] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3524.093725] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3526.829483] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3535.361425] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3539.213491] Lustre: 117870:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3592.323452] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 09:59:19 (1789480759) [ 3621.349797] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3621.357973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3621.371077] Lustre: Skipped 3 previous similar messages [ 3621.393689] Lustre: Skipped 11 previous similar messages [ 3625.600303] Lustre: server umount lustre-MDT0000 complete [ 3630.123630] LustreError: 115993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789480799 with bad export cookie 13937112777164109449 [ 3630.136904] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3630.141511] LustreError: 115993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3636.876551] Lustre: server umount lustre-MDT0001 complete [ 3647.439982] Lustre: server umount lustre-OST0000 complete [ 3658.293649] Lustre: server umount lustre-OST0001 complete [ 3665.590576] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 3676.031658] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 3716.980392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3717.251193] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3717.274789] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3717.365279] Lustre: lustre-MDT0000: new disk, initializing [ 3717.494693] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3722.482185] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3730.152111] Lustre: server umount lustre-MDT0000 complete [ 3742.126657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3742.431885] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3742.449270] Lustre: 122152:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3742.463981] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3742.737509] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3742.740069] Lustre: Skipped 4 previous similar messages [ 3748.513455] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3753.650410] Lustre: server umount lustre-MDT0000 complete [ 3763.563420] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 10:02:10 (1789480930) [ 3765.432868] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3767.615635] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 10:02:14 (1789480934) [ 3786.752514] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 3797.237660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3802.263210] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3811.560178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3811.742313] Lustre: 124201:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3811.799766] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3811.809457] Lustre: Skipped 1 previous similar message [ 3811.914866] Lustre: lustre-MDT0001: new disk, initializing [ 3812.091522] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3812.102833] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3816.128102] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3820.953670] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3827.259075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3827.508160] Lustre: lustre-OST0000: new disk, initializing [ 3827.514890] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3827.520834] Lustre: 125780:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3829.310778] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3829.320514] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3829.441504] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3835.013121] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3846.441067] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3846.703557] Lustre: lustre-OST0001: new disk, initializing [ 3846.706897] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3846.713869] Lustre: 126657:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3848.578243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3848.585695] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3848.696547] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3853.589234] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3863.479994] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3866.501184] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4236.083341] Lustre: 133069:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4236.092548] Lustre: 133069:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4236.100425] Lustre: 133069:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4236.106348] Lustre: 133069:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4236.110774] Lustre: 133069:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4236.115516] Lustre: 133069:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4490.100159] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 10:14:17 (1789481657) [ 4504.547686] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4504.554818] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4504.566527] Lustre: Skipped 2 previous similar messages [ 4504.583412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4504.587801] Lustre: Skipped 4 previous similar messages [ 4509.154045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4509.156744] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4509.177926] Lustre: Skipped 2 previous similar messages [ 4511.004040] Lustre: server umount lustre-MDT0000 complete [ 4514.278396] LustreError: 127468:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4514.303949] LustreError: 127468:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 22 previous similar messages [ 4514.473599] LustreError: 126656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789481683 with bad export cookie 13937112777164177748 [ 4514.475127] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4514.487674] LustreError: 126656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4519.394087] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4519.404651] Lustre: Skipped 1 previous similar message [ 4519.419720] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4519.427069] Lustre: Skipped 4 previous similar messages [ 4521.073699] Lustre: server umount lustre-MDT0001 complete [ 4530.553478] Lustre: server umount lustre-OST0000 complete [ 4539.776627] Lustre: server umount lustre-OST0001 complete [ 4546.165228] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 4553.612952] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 4582.611074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4582.846421] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4582.865204] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4582.947403] Lustre: lustre-MDT0000: new disk, initializing [ 4583.015447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4583.020566] Lustre: Skipped 4 previous similar messages [ 4583.040708] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4586.794224] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4595.313783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4595.510755] Lustre: 136115:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4595.580845] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4595.587392] Lustre: Skipped 1 previous similar message [ 4595.649820] Lustre: lustre-MDT0001: new disk, initializing [ 4595.770270] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4595.781075] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4601.468461] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4616.102638] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4624.172722] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4633.882918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4634.486132] Lustre: lustre-OST0000: new disk, initializing [ 4634.493263] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4634.501562] Lustre: 138085:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4636.224873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4636.248848] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4636.498141] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4645.123917] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4659.791762] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4671.393447] Lustre: lustre-MDD0000: changelog on [ 4674.898777] Lustre: lustre-MDD0001: changelog on [ 6674.292961] Lustre: lustre-MDD0001: changelog off [ 6676.928533] Lustre: 136123:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6678.139973] Lustre: lustre-MDD0000: changelog off [ 6689.762967] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6689.776271] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6689.788864] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6689.795031] Lustre: Skipped 1 previous similar message [ 6691.376748] Lustre: server umount lustre-OST0000 complete [ 6700.005896] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6700.017635] Lustre: Skipped 1 previous similar message [ 6700.017800] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6700.024189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6700.047583] LustreError: Skipped 1 previous similar message [ 6702.872683] Lustre: server umount lustre-MDT0000 complete [ 6705.127608] LustreError: 138027:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6705.164797] LustreError: 138027:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6707.552940] LustreError: 136106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789483876 with bad export cookie 13937112777164997756 [ 6707.563755] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6707.574198] LustreError: 136106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6708.061943] Lustre: server umount lustre-MDT0001 complete [ 6720.204777] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 6723.497662] Key type lgssc unregistered [ 6723.718239] LNet: 147244:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6723.723838] LNetError: 147244:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6724.791541] LNet: Removed LNI 192.168.202.133@tcp [ 6725.720520] Key type .llcrypt unregistered [ 6725.722470] Key type ._llcrypt unregistered [ 6748.054485] Key type ._llcrypt registered [ 6748.059992] Key type .llcrypt registered [ 6748.278733] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 6763.447710] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 6764.676816] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6764.835564] alg: No test for adler32 (adler32-zlib) [ 6766.106423] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [ 6766.371120] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 6768.036041] Key type lgssc registered [ 6769.256833] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6810.900962] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6810.917987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6812.431378] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6812.487878] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6812.659757] Lustre: lustre-MDT0000: new disk, initializing [ 6812.767894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6812.782704] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6818.197429] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6829.844282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6829.985514] Lustre: 150458:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6830.053954] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6830.057815] Lustre: Skipped 1 previous similar message [ 6830.134912] Lustre: lustre-MDT0001: new disk, initializing [ 6830.233892] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6830.285588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6830.296490] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6835.344732] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6845.717141] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6851.717407] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6859.085905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6859.500406] Lustre: lustre-OST0000: new disk, initializing [ 6859.513139] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6859.526695] Lustre: 152425:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6859.629966] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6866.854478] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6868.614542] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6868.653928] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6868.799821] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6878.332512] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6884.852760] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_module obdecho/obdecho [ 6886.784450] LustreError: 153429:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6886.796407] LustreError: 153429:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6888.837208] LustreError: 153525:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6902.244819] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6902.256189] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6902.280342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6904.805535] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6904.815540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6904.833477] Lustre: Skipped 1 previous similar message [ 6904.848386] Lustre: Skipped 2 previous similar messages [ 6907.073520] Lustre: server umount lustre-MDT0000 complete [ 6911.342874] LustreError: 150449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789484080 with bad export cookie 9454668394962432291 [ 6911.349158] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6911.359166] LustreError: 150449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6911.701363] Lustre: server umount lustre-MDT0001 complete [ 6925.942555] Lustre: server umount lustre-OST0000 complete [ 6947.279698] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 6957.593221] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 6996.618196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6997.033803] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6997.100303] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6997.215187] Lustre: lustre-MDT0000: new disk, initializing [ 6997.317080] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6997.337106] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7002.031293] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7012.131969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7012.199735] Lustre: 157243:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7012.239772] LustreError: 157253:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7012.255569] LustreError: 157253:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 7012.266792] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7012.269138] Lustre: Skipped 1 previous similar message [ 7012.336622] Lustre: lustre-MDT0001: new disk, initializing [ 7012.413532] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7012.445830] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7012.458825] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7018.353781] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7032.369100] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7038.939348] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7046.755178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7047.039161] Lustre: lustre-OST0000: new disk, initializing [ 7047.043146] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7047.051776] Lustre: 159210:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7047.188912] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7052.729296] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7055.452997] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7055.476533] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7055.701578] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7061.706712] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7067.134912] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7067.367638] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7073.180958] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7073.512209] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7076.321164] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7076.330874] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7076.331717] LustreError: Skipped 1 previous similar message [ 7076.334156] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7076.349253] Lustre: Skipped 1 previous similar message [ 7081.441460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7081.460391] Lustre: Skipped 2 previous similar messages [ 7086.592317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7086.615057] Lustre: Skipped 1 previous similar message [ 7090.144868] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7090.604313] Lustre: server umount lustre-OST0000 complete [ 7096.808399] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7096.818797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7096.820342] Lustre: Skipped 1 previous similar message [ 7096.837052] Lustre: Skipped 1 previous similar message [ 7110.113984] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7110.378622] Lustre: server umount lustre-MDT0000 complete [ 7112.167674] LustreError: 157254:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7112.188558] LustreError: 157254:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7115.380457] LustreError: 159209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789484284 with bad export cookie 9454668394962433243 [ 7115.382581] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7115.392934] LustreError: 159209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7115.821588] Lustre: server umount lustre-MDT0001 complete [ 7119.727945] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 10:58:07 (1789484287) [ 7121.398992] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7123.393485] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 10:58:10 (1789484290) [ 7131.176477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7131.475747] LustreError: 161627:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7131.540505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7135.829116] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7136.748808] LustreError: 161628:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7141.860527] LustreError: 161627:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7144.613155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7150.352804] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7158.416974] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7164.564572] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7171.524898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7171.822976] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7171.833816] Lustre: Skipped 1 previous similar message [ 7178.549529] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7186.337970] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7197.175295] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.33@tcp (stopping) [ 7197.180488] Lustre: Skipped 4 previous similar messages [ 7198.507977] Lustre: server umount lustre-MDT0000 complete [ 7201.762346] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7201.769863] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7201.771178] LustreError: 161622:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7201.818302] Lustre: Skipped 2 previous similar messages [ 7202.243886] LustreError: 161608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789484371 with bad export cookie 9454668394962434132 [ 7202.249693] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7202.267261] LustreError: 161608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7208.613904] Lustre: server umount lustre-MDT0001 complete [ 7216.252589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7231.610808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7231.924611] LustreError: 165469:0:(llog.c:1655:llog_backup()) MGC192.168.202.133@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7231.933783] Lustre: 165469:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.133@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7232.492400] LustreError: 165474:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7232.519367] LustreError: 165474:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 7268.324271] LustreError: 165493:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7268.341714] LustreError: 165493:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 29 previous similar messages [ 7292.897351] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7292.913244] Lustre: 165469:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.133@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7333.878293] LustreError: 165474:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7333.894277] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7333.901280] LustreError: 165474:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 41 previous similar messages [ 7338.948642] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7338.983379] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7348.253818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7362.534075] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7378.349284] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7378.433532] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7385.679622] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7387.499840] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7393.233650] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7394.782560] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7414.245791] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7414.247964] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7414.272084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7414.287271] Lustre: Skipped 2 previous similar messages [ 7414.310121] Lustre: Skipped 6 previous similar messages [ 7418.224970] Lustre: server umount lustre-MDT0000 complete [ 7421.769822] LustreError: 161609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789484590 with bad export cookie 9454668394962440621 [ 7421.776822] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7421.793769] LustreError: 161609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7422.241288] Lustre: server umount lustre-MDT0001 complete [ 7436.539988] Lustre: server umount lustre-OST0000 complete [ 7460.635465] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 7471.049771] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 7510.087408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7510.374692] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7510.413753] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7510.512398] Lustre: lustre-MDT0000: new disk, initializing [ 7510.594071] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7510.602292] Lustre: Skipped 1 previous similar message [ 7510.614910] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7515.711267] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7526.896904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7527.115951] Lustre: 170598:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7527.549281] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7532.734493] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7541.675165] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7547.048650] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7553.930393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7554.156904] Lustre: lustre-OST0000: new disk, initializing [ 7554.160492] Lustre: Skipped 1 previous similar message [ 7554.163718] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7554.166022] Lustre: Skipped 2 previous similar messages [ 7554.168760] Lustre: 172567:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7561.723775] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7564.343418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7564.354969] Lustre: Skipped 1 previous similar message [ 7564.363249] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7564.501352] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7574.049813] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7579.581901] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7579.841522] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7586.118214] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7586.461769] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7590.369886] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7590.381419] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7590.389880] LustreError: Skipped 1 previous similar message [ 7590.394692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7590.466371] Lustre: Skipped 2 previous similar messages [ 7595.247609] Lustre: server umount lustre-OST0000 complete [ 7605.553455] Lustre: server umount lustre-MDT0000 complete [ 7605.744656] LustreError: 171880:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7605.772820] LustreError: 171880:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [ 7611.062651] LustreError: 171392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789484780 with bad export cookie 9454668394962457421 [ 7611.078388] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7611.101517] LustreError: 171392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7611.517345] Lustre: server umount lustre-MDT0001 complete [ 7622.907720] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 11:06:29 (1789484789) [ 7631.973429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7638.527697] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7649.337769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7654.797617] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7663.703456] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7670.444518] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7677.570433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7685.248401] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7692.724857] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7714.619950] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 11:08:01 (1789484881) [ 7719.908084] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7719.911462] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7719.925412] Lustre: Skipped 2 previous similar messages [ 7719.928452] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7719.930753] Lustre: Skipped 3 previous similar messages [ 7725.884216] Lustre: server umount lustre-OST0000 complete [ 7736.814748] Lustre: server umount lustre-MDT0000 complete [ 7740.843959] LustreError: 174961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789484909 with bad export cookie 9454668394962458303 [ 7740.870965] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7740.883543] LustreError: 174961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7741.240851] Lustre: server umount lustre-MDT0001 complete [ 7754.310951] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 7757.662692] Key type lgssc unregistered [ 7757.902987] LNet: 178737:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7757.921860] LNetError: 178737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7757.941988] LNet: Removed LNI 192.168.202.133@tcp [ 7758.995887] Key type .llcrypt unregistered [ 7758.999585] Key type ._llcrypt unregistered [ 7780.065975] Key type ._llcrypt registered [ 7780.068890] Key type .llcrypt registered [ 7780.269768] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 7781.816505] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7781.888236] alg: No test for adler32 (adler32-zlib) [ 7783.317214] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [ 7783.630745] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 7785.272750] Key type lgssc registered [ 7787.214260] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7797.000462] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7797.013762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7812.512836] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7817.634240] LustreError: 179981:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.133@tcp: failed processing log, type 4: rc = -110 [ 7822.752463] LustreError: 179981:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7822.765103] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7822.773736] LustreError: MGC192.168.202.133@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7822.787581] LustreError: 179981:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7822.802474] LustreError: 179981:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -22 [ 7822.821560] LustreError: 179981:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 7822.902593] Lustre: server umount lustre-OST0000 complete [ 7822.912796] LustreError: 179981:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7831.532259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7832.965151] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7837.907245] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7842.283237] LustreError: 180480:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7842.300531] LustreError: 180480:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7847.421016] LustreError: 180481:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7848.281215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7848.678744] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7853.783902] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7861.255629] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7867.334371] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7875.188036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7875.350630] LustreError: 182296:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7875.357604] Lustre: 182296:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.202.133@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7875.490345] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7883.906224] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7892.992711] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7899.110790] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7899.114596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7899.141205] Lustre: Skipped 2 previous similar messages [ 7899.160693] Lustre: Skipped 2 previous similar messages [ 7901.812753] Lustre: server umount lustre-MDT0000 complete [ 7904.231482] LustreError: 180477:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7904.256106] LustreError: 180477:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7905.711500] LustreError: 183058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789485074 with bad export cookie 7994706424792774939 [ 7905.722884] LustreError: 183058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7905.733217] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7906.094494] Lustre: server umount lustre-MDT0001 complete [ 7919.237367] Lustre: server umount lustre-OST0000 complete [ 7929.562418] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 7940.893461] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 7982.571109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7982.882879] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7982.937531] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7983.102589] Lustre: lustre-MDT0000: new disk, initializing [ 7983.179898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7983.199281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7988.326665] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7999.612394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7999.823613] Lustre: 185824:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7999.918581] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7999.922129] Lustre: Skipped 1 previous similar message [ 8000.080954] Lustre: lustre-MDT0001: new disk, initializing [ 8000.280077] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8000.318941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8000.339186] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8006.426599] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 8017.351603] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8023.597140] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8028.415303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8028.635287] Lustre: lustre-OST0000: new disk, initializing [ 8028.638204] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8028.642921] Lustre: 187792:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8028.721970] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8033.573369] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 8034.347366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8034.376090] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8034.578195] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8044.556619] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8051.130710] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8051.360586] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8057.557498] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8057.775652] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8065.509971] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8065.526248] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8065.533038] LustreError: Skipped 1 previous similar message [ 8065.534900] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8065.549178] Lustre: Skipped 1 previous similar message [ 8065.584361] Lustre: Skipped 1 previous similar message [ 8066.730427] Lustre: server umount lustre-OST0000 complete [ 8075.759991] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8075.780386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8075.797372] Lustre: Skipped 1 previous similar message [ 8075.828984] Lustre: Skipped 1 previous similar message [ 8079.727393] Lustre: server umount lustre-MDT0000 complete [ 8080.875237] LustreError: 185834:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8080.897073] LustreError: 185834:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8084.996085] LustreError: 185834:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8085.039135] LustreError: 185834:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8086.899736] LustreError: 185815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789485255 with bad export cookie 7994706424792775667 [ 8086.905549] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8086.922043] LustreError: 185815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8088.280747] Lustre: server umount lustre-MDT0001 complete [ 8102.559439] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 11:14:28 (1789485268) [ 8122.525870] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 8124.691239] Key type lgssc unregistered [ 8124.871592] LNet: 190744:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8124.874885] LNetError: 190744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8124.888381] LNet: Removed LNI 192.168.202.133@tcp [ 8125.556123] Key type .llcrypt unregistered [ 8125.559650] Key type ._llcrypt unregistered [ 8144.022135] Key type ._llcrypt registered [ 8144.028051] Key type .llcrypt registered [ 8144.096571] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 8145.245239] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8145.280041] alg: No test for adler32 (adler32-zlib) [ 8146.638827] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [ 8147.022077] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 8148.786516] Key type lgssc registered [ 8150.588824] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8159.152614] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8159.168915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8160.714220] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8166.334436] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 8170.987161] LustreError: 191961:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8171.023803] LustreError: 191961:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8174.054744] LustreError: 191960:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8175.622820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8176.015571] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8179.670192] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 8187.221916] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8193.345629] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8201.808589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8201.974102] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8201.980200] LustreError: 193823:0:(llog.c:1696:llog_backup()) MGC192.168.202.133@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8201.985643] Lustre: 193823:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.202.133@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8202.134865] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8208.364908] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 8216.584198] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8220.901892] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8220.912154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8220.920874] Lustre: Skipped 2 previous similar messages [ 8220.943501] Lustre: Skipped 2 previous similar messages [ 8225.200680] Lustre: server umount lustre-MDT0000 complete [ 8226.274435] LustreError: 191960:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8226.291290] LustreError: 191960:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8228.696680] LustreError: 191940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789485397 with bad export cookie 5634096245391535242 [ 8228.705487] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8228.709463] LustreError: 191940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8229.112289] LustreError: 192689:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000ca114102 x1876411354738560/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8229.266843] Lustre: server umount lustre-MDT0001 complete [ 8243.284380] Lustre: server umount lustre-OST0000 complete [ 8253.339396] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 8262.796545] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 8305.698225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8305.858437] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8305.884408] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8305.972611] Lustre: lustre-MDT0000: new disk, initializing [ 8306.065042] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8306.078214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8309.977080] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 8321.848228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8322.029469] Lustre: 197353:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8322.083190] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8322.084482] Lustre: Skipped 1 previous similar message [ 8322.085929] LustreError: 198135:0:(llog.c:1696:llog_backup()) MGC192.168.202.133@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8322.096038] LustreError: 198135:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8322.100147] Lustre: 198135:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.133@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8322.134536] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8322.143825] Lustre: Skipped 1 previous similar message [ 8322.282859] Lustre: lustre-MDT0001: new disk, initializing [ 8322.418569] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8322.461545] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8322.476816] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8329.027128] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 8340.373056] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8347.368725] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8354.370273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8354.590135] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8354.591224] Lustre: Skipped 1 previous similar message [ 8354.592376] LustreError: 199298:0:(llog.c:1696:llog_backup()) MGC192.168.202.133@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8354.594817] LustreError: 199298:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8354.597460] Lustre: 199298:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.133@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8354.601021] Lustre: 199298:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8354.613485] Lustre: lustre-OST0000: new disk, initializing [ 8354.616089] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8354.619295] Lustre: 199323:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8354.670825] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8361.110372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8361.134052] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8361.307155] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8362.093683] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 8373.532745] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8379.037332] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8379.265126] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8385.101485] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8385.510639] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8392.170167] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8392.176628] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8392.181411] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8392.190503] Lustre: Skipped 1 previous similar message [ 8397.286896] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8397.294686] Lustre: Skipped 2 previous similar messages [ 8402.403945] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8402.420117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8402.466292] Lustre: Skipped 1 previous similar message [ 8402.703601] Lustre: server umount lustre-OST0000 complete [ 8409.571686] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8409.589636] LustreError: Skipped 1 previous similar message [ 8409.597581] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8409.626196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8412.644029] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8417.764377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8417.767979] Lustre: Skipped 3 previous similar messages [ 8422.368138] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8422.878946] Lustre: server umount lustre-MDT0000 complete [ 8422.885934] LustreError: 197364:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8422.919433] LustreError: 197364:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8427.403254] LustreError: 197343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789485596 with bad export cookie 5634096245391535956 [ 8427.405985] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8427.416556] LustreError: 197343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8427.715959] Lustre: server umount lustre-MDT0001 complete [ 8437.051822] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 11:20:04 (1789485604) [ 8454.456943] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 8464.634659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8465.080882] LustreError: 202455:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8465.139222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8469.786667] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 8475.623989] LustreError: 202456:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8475.649865] LustreError: 202456:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8479.292109] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8483.923767] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 8486.564762] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8493.668475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8500.119308] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 8508.474835] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8508.595706] Lustre: lustre-OST0001: new disk, initializing [ 8508.598407] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8508.614488] Lustre: 204762:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8508.676349] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8508.681199] Lustre: Skipped 2 previous similar messages [ 8513.633435] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 8518.399288] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8518.409776] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8518.541705] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8526.428739] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8536.746687] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8540.304262] Lustre: DEBUG MARKER: ADD OST3 [ 8547.243164] loop: module loaded [ 8547.314102] loop0: detected capacity change from 0 to 204800000 [ 8547.574812] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8547.604444] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8547.630641] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8554.502062] loop0: detected capacity change from 0 to 204800000 [ 8555.304950] Lustre: DEBUG MARKER: STOP OST3 [ 8560.800301] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8561.021966] Lustre: lustre-OST0003: new disk, initializing [ 8561.025898] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8561.036845] Lustre: 206762:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8562.253924] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8562.433485] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8562.441649] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8562.500945] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8564.321326] LustreError: 206885:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8565.129972] Lustre: Failing over lustre-OST0003 [ 8565.288052] Lustre: server umount lustre-OST0003 complete [ 8567.793874] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8567.798910] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8567.811817] LustreError: 203943:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8567.827926] Lustre: Skipped 1 previous similar message [ 8567.857032] LustreError: 203943:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8567.930330] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 8578.786350] Lustre: DEBUG MARKER: START OST3 again [ 8587.419040] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8587.655277] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8587.668129] Lustre: Skipped 1 previous similar message [ 8587.686739] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8588.239284] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8589.598307] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8589.603455] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8593.749031] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 8614.508418] Lustre: Failing over lustre-OST0003 [ 8614.723212] Lustre: server umount lustre-OST0003 complete [ 8615.395312] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8615.398213] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8615.405701] LustreError: Skipped 1 previous similar message [ 8615.406862] LustreError: 203943:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8615.406869] LustreError: 203943:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 8615.468628] Lustre: Skipped 1 previous similar message [ 8633.313039] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8633.316409] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8633.332132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8639.738900] Lustre: server umount lustre-MDT0000 complete [ 8644.129225] LustreError: 203191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789485813 with bad export cookie 5634096245391536838 [ 8644.144314] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8644.151261] LustreError: 203191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8650.839534] Lustre: server umount lustre-MDT0001 complete [ 8655.178888] Lustre: server umount lustre-OST0000 complete [ 8659.976247] Lustre: server umount lustre-OST0001 complete [ 8668.967031] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 8679.816636] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 8718.243826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8718.524285] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8718.556129] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8718.719496] Lustre: lustre-MDT0000: new disk, initializing [ 8718.800732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8718.817710] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8723.253744] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 8733.751287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8733.881182] Lustre: 212225:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8734.214334] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8739.541340] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 8749.188464] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8755.322141] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8763.163601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8763.517240] Lustre: lustre-OST0000: new disk, initializing [ 8763.529421] Lustre: Skipped 1 previous similar message [ 8763.533772] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8763.549370] Lustre: Skipped 2 previous similar messages [ 8763.554629] Lustre: 214196:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8764.808029] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8764.826129] Lustre: Skipped 1 previous similar message [ 8764.839928] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8764.961639] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8769.609430] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 8779.343630] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8783.806680] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8784.015778] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8788.950299] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8789.202416] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8795.623740] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8795.624242] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8795.625960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8795.625966] Lustre: Skipped 9 previous similar messages [ 8795.631103] LustreError: Skipped 1 previous similar message [ 8795.665957] Lustre: Skipped 6 previous similar messages [ 8805.344244] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8805.454547] Lustre: server umount lustre-OST0000 complete [ 8823.264591] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8823.520770] Lustre: server umount lustre-MDT0000 complete [ 8825.314907] LustreError: 213845:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8825.347777] LustreError: 213845:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 17 previous similar messages [ 8828.125383] LustreError: 213020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789485997 with bad export cookie 5634096245391539141 [ 8828.128938] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8828.153548] LustreError: 213020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8828.768919] Lustre: server umount lustre-MDT0001 complete [ 8832.767710] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 11:26:40 (1789486000) [ 8849.307845] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 8860.093527] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 8902.537759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8902.890694] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8902.964253] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8903.021625] Lustre: lustre-MDT0000: new disk, initializing [ 8903.118917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8907.617881] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 8919.602537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8919.763912] Lustre: 218806:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8920.143275] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8925.315348] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 8935.319927] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8942.216602] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8949.674958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8950.075660] Lustre: 220780:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8951.996342] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8956.821084] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 8966.342854] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8971.802211] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8972.162065] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8977.455189] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8977.692358] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8982.498436] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8982.512855] LustreError: Skipped 1 previous similar message [ 8982.517964] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8982.536317] Lustre: Skipped 2 previous similar messages [ 8982.548615] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8982.555511] Lustre: Skipped 9 previous similar messages [ 8985.824790] Lustre: server umount lustre-OST0000 complete [ 8996.137202] Lustre: server umount lustre-MDT0000 complete [ 8998.376257] LustreError: 218819:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8998.414424] LustreError: 218819:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9000.929327] LustreError: 219604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789486169 with bad export cookie 5634096245391540023 [ 9000.937124] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9000.943574] LustreError: 219604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9001.385445] Lustre: server umount lustre-MDT0001 complete [ 9009.838990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9010.248296] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9010.252500] Lustre: Skipped 5 previous similar messages [ 9015.452423] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 9024.580268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9028.930143] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 9037.526921] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9043.598623] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9050.609947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9056.589272] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 9065.405954] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9079.777614] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9079.793084] LustreError: Skipped 2 previous similar messages [ 9082.743199] Lustre: server umount lustre-OST0000 complete [ 9094.893100] Lustre: server umount lustre-MDT0000 complete [ 9099.516905] LustreError: 222975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789486268 with bad export cookie 5634096245391540898 [ 9099.529806] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9099.836906] Lustre: server umount lustre-MDT0001 complete [ 9207.671883] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 9210.899845] Key type lgssc unregistered [ 9211.290514] LNet: 226662:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9211.295672] LNetError: 226662:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9212.327650] LNet: Removed LNI 192.168.202.133@tcp [ 9213.370208] Key type .llcrypt unregistered [ 9213.373794] Key type ._llcrypt unregistered [ 9246.098285] Key type ._llcrypt registered [ 9246.100281] Key type .llcrypt registered [ 9246.192463] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 9263.484330] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 9265.022726] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9265.098588] alg: No test for adler32 (adler32-zlib) [ 9266.377896] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [ 9266.782769] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 9268.536915] Key type lgssc registered [ 9269.727623] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9313.900868] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9313.980144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9315.155450] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9315.187956] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9315.246220] Lustre: lustre-MDT0000: new disk, initializing [ 9315.302538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9315.325231] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9319.813513] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 9331.083906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9331.210911] Lustre: 230019:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9331.259631] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9331.262771] Lustre: Skipped 1 previous similar message [ 9331.348134] Lustre: lustre-MDT0001: new disk, initializing [ 9331.473580] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9331.554211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9331.593848] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9337.305314] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 9347.830453] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9353.987117] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9362.014205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9362.279821] Lustre: lustre-OST0000: new disk, initializing [ 9362.283189] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9362.289781] Lustre: 231987:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9362.392770] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9368.646488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9368.659277] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9368.783957] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9369.088679] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 9379.408329] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9385.418947] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9385.682600] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9390.677516] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9390.870381] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9394.671347] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9394.673065] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9394.683233] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9394.699082] Lustre: Skipped 1 previous similar message [ 9398.996566] Lustre: server umount lustre-OST0000 complete [ 9403.370357] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9403.378383] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9403.399617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9403.402283] Lustre: Skipped 1 previous similar message [ 9404.901862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9404.912076] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9410.141383] Lustre: server umount lustre-MDT0000 complete [ 9414.237575] LustreError: 230011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789486583 with bad export cookie 17041554548507954212 [ 9414.246050] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9414.249096] LustreError: 230011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9414.548157] Lustre: server umount lustre-MDT0001 complete [ 9436.161543] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 9447.116125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9447.385684] LustreError: 234922:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9447.401436] LustreError: 234922:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9447.428104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9451.801692] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 9452.539792] LustreError: 234923:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9457.648397] LustreError: 234922:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9462.043948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9462.352835] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9468.027501] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 9471.681103] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9480.076340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9480.549919] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9488.630819] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 9499.362881] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9499.588324] Lustre: lustre-OST0001: new disk, initializing [ 9499.592574] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9499.602548] Lustre: 237228:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9499.779102] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9507.351578] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 9507.987179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9507.999863] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9508.135244] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9519.625783] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9527.558245] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9541.836804] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 11:38:28 (1789486708) [ 9549.298114] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9549.315845] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9549.317420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9549.317426] Lustre: Skipped 1 previous similar message [ 9549.329338] Lustre: Skipped 2 previous similar messages [ 9550.721467] Lustre: server umount lustre-MDT0000 complete [ 9554.403745] LustreError: 234918:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9554.434950] LustreError: 234918:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9555.452685] LustreError: 234902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789486724 with bad export cookie 17041554548507955094 [ 9555.463125] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9555.477672] LustreError: 234902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9555.939850] Lustre: server umount lustre-MDT0001 complete [ 9570.986417] Lustre: server umount lustre-OST0000 complete [ 9586.041652] Lustre: server umount lustre-OST0001 complete [ 9594.805511] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 9608.520818] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 9648.672103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9648.904093] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9648.931655] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9648.980953] Lustre: lustre-MDT0000: new disk, initializing [ 9649.037543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9649.050975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9653.677742] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 9667.110664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9667.370683] Lustre: 241180:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9667.483924] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9667.494658] Lustre: Skipped 1 previous similar message [ 9667.640350] Lustre: lustre-MDT0001: new disk, initializing [ 9667.825581] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9667.840363] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9674.256965] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 9686.647296] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9694.960656] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9703.519266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9703.945731] Lustre: lustre-OST0000: new disk, initializing [ 9703.953087] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9703.963064] Lustre: 243151:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9705.989690] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9706.007799] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9706.181417] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9711.966625] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 9722.641819] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9728.308594] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9728.619364] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9733.668296] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9733.984393] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9736.676247] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9736.681970] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9736.692613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9736.702164] Lustre: Skipped 4 previous similar messages [ 9742.617314] Lustre: server umount lustre-OST0000 complete [ 9749.986156] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9749.995789] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9750.007185] Lustre: Skipped 1 previous similar message [ 9750.015403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9750.026830] Lustre: Skipped 2 previous similar messages [ 9762.275776] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9762.602672] Lustre: server umount lustre-MDT0000 complete [ 9766.381023] LustreError: 241186:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9766.418091] LustreError: 241186:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9767.227195] LustreError: 241170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789486936 with bad export cookie 17041554548507957229 [ 9767.231942] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9767.244694] LustreError: 241170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9767.561609] Lustre: server umount lustre-MDT0001 complete [ 9777.084851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9777.489219] LustreError: 245367:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9777.594229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9777.603175] Lustre: Skipped 2 previous similar messages [ 9781.918752] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 9792.080983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9797.289832] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 9804.892801] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9811.116131] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9819.199814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9827.149754] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 9837.248585] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9985.515379] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9985.522528] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9985.527493] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9985.532781] Lustre: Skipped 2 previous similar messages [ 9985.538369] Lustre: Skipped 7 previous similar messages [ 9989.405405] Lustre: server umount lustre-OST0000 complete [ 9999.732975] Lustre: server umount lustre-MDT0000 complete [10000.870052] LustreError: 246372:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10000.885757] LustreError: 246372:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10004.174409] LustreError: 245348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789487173 with bad export cookie 17041554548507958104 [10004.184104] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10004.193777] LustreError: 245348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10004.676317] Lustre: server umount lustre-MDT0001 complete [10108.793074] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [10112.018444] Key type lgssc unregistered [10112.230940] LNet: 248986:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10112.237454] LNetError: 248986:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10112.250924] LNet: Removed LNI 192.168.202.133@tcp [10113.037235] Key type .llcrypt unregistered [10113.040227] Key type ._llcrypt unregistered [10143.879714] Key type ._llcrypt registered [10143.882979] Key type .llcrypt registered [10144.007223] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [10160.912356] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [10162.199566] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10162.328904] alg: No test for adler32 (adler32-zlib) [10163.605844] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [10163.955087] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [10165.704189] Key type lgssc registered [10167.135739] Lustre: Echo OBD driver; http://www.lustre.org/ [10215.062677] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [10225.985758] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10226.018777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10227.494519] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10227.550745] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10227.690270] Lustre: lustre-MDT0000: new disk, initializing [10227.840338] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10227.863458] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10232.811580] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [10244.579042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10244.739589] Lustre: 253056:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10244.855719] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10244.858213] Lustre: Skipped 1 previous similar message [10244.976508] Lustre: lustre-MDT0001: new disk, initializing [10245.136687] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10245.195810] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10245.205650] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10249.907286] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [10255.383866] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10262.627716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10263.030786] Lustre: lustre-OST0000: new disk, initializing [10263.034710] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10263.040906] Lustre: 254689:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10263.180835] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10269.709072] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [10270.316680] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10270.330840] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10270.451160] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10279.884613] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10280.025679] Lustre: lustre-OST0001: new disk, initializing [10280.034051] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10280.046741] Lustre: 255557:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10280.152743] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10286.018746] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [10289.262747] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10289.266069] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10289.346795] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10298.791549] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10306.452795] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10320.368383] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10320.384728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10320.399186] Lustre: Skipped 3 previous similar messages [10320.404225] Lustre: Skipped 2 previous similar messages [10324.831531] Lustre: server umount lustre-MDT0000 complete [10329.693978] LustreError: 253046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789487498 with bad export cookie 12642100438916511282 [10329.712096] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10329.722595] LustreError: 253046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10330.164841] Lustre: server umount lustre-MDT0001 complete [10345.595572] Lustre: server umount lustre-OST0000 complete [10346.464916] Lustre: 250561:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789487499/real 1789487499] req@ffff8909439fa680 x1876413468716288/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789487515 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10346.513327] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10349.353789] Lustre: server umount lustre-OST0001 complete [10364.127866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10374.600253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10384.419631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10395.552054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10417.778110] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [10428.818473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10428.833915] Lustre: lustre-MDT0000: reset Object Index mappings [10429.398157] LustreError: 260571:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10429.417562] LustreError: 260571:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10429.484119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10433.834833] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [10434.539290] LustreError: 260572:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10439.650090] LustreError: 260571:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10444.775238] LustreError: 260572:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10444.942427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10444.977413] Lustre: lustre-MDT0001: reset Object Index mappings [10445.629304] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10451.130821] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [10455.667957] Lustre: 261713:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10463.467054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10464.089795] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10470.254390] LustreError: 262068:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10471.322512] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [10474.382089] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10479.596164] LustreError: 262069:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10479.634232] LustreError: 262069:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10480.558517] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10486.300135] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10488.393571] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [10497.013536] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10500.936407] Lustre: 263585:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10506.725804] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10506.739330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10506.747786] Lustre: Skipped 3 previous similar messages [10506.754662] Lustre: Skipped 3 previous similar messages [10511.844969] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10513.078376] Lustre: server umount lustre-MDT0000 complete [10516.963443] LustreError: 262587:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10516.980299] LustreError: 262587:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10517.230197] LustreError: 260553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789487686 with bad export cookie 12642100438916518072 [10517.236201] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10517.239256] LustreError: 260553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10517.551937] Lustre: server umount lustre-MDT0001 complete [10532.615186] Lustre: server umount lustre-OST0000 complete [10546.489710] Lustre: server umount lustre-OST0001 complete [10566.898458] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 11:55:34 (1789487734) [10584.243899] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [10594.718232] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [10637.692161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10638.034632] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10638.070677] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10638.192465] Lustre: lustre-MDT0000: new disk, initializing [10638.306614] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10638.311124] Lustre: Skipped 1 previous similar message [10638.337829] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10643.495602] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [10655.610431] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10655.783427] Lustre: 267840:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10655.949606] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10655.962951] Lustre: Skipped 1 previous similar message [10656.099936] Lustre: lustre-MDT0001: new disk, initializing [10656.336169] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10656.359924] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10662.146168] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [10673.241722] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10681.144573] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10690.600523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10691.021785] Lustre: lustre-OST0000: new disk, initializing [10691.024941] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10691.039809] Lustre: 269811:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10693.091480] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10693.115705] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10693.243136] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10699.241873] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [10710.201503] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10716.279669] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10716.632055] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10721.838733] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10722.106411] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10728.931226] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10728.945494] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10728.955734] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10728.961608] Lustre: Skipped 4 previous similar messages [10730.568936] Lustre: server umount lustre-OST0000 complete [10738.151301] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10738.160079] LustreError: Skipped 1 previous similar message [10738.166318] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10738.185851] Lustre: Skipped 1 previous similar message [10738.205290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10741.612568] Lustre: server umount lustre-MDT0000 complete [10744.289934] LustreError: 267846:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10744.300872] LustreError: 267846:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10745.981386] LustreError: 267832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789487915 with bad export cookie 12642100438916520165 [10745.983471] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10745.995540] LustreError: 267832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10746.391971] Lustre: server umount lustre-MDT0001 complete [10768.180588] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [10779.349170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10779.720583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10779.726409] Lustre: Skipped 2 previous similar messages [10785.535180] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [10795.089891] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10800.589582] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [10804.095422] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10814.585478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10821.760508] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [10830.699677] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10830.961216] Lustre: lustre-OST0001: new disk, initializing [10830.965443] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10830.973088] Lustre: 275057:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10832.890760] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10832.897643] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10832.987469] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10837.561223] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [10848.431695] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10852.469746] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10858.516868] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10858.838963] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10867.175639] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10867.184373] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10867.203153] Lustre: Skipped 1 previous similar message [10867.210953] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10867.215241] Lustre: Skipped 2 previous similar messages [10871.450948] Lustre: server umount lustre-MDT0000 complete [10873.837353] LustreError: 272749:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10873.863907] LustreError: 272749:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [10876.003280] LustreError: 272729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789488045 with bad export cookie 12642100438916521047 [10876.009981] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10876.014957] LustreError: 272729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10876.689957] Lustre: server umount lustre-MDT0001 complete [10891.007760] Lustre: server umount lustre-OST0000 complete [10894.947731] Lustre: 250562:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789488047/real 1789488047] req@ffff89094d4e9c00 x1876413468877824/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789488063 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10894.978450] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10894.996383] Lustre: Skipped 3 previous similar messages [10896.290906] Lustre: server umount lustre-OST0001 complete [10914.154216] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [10924.206431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10924.579204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10924.585214] Lustre: Skipped 3 previous similar messages [10930.568286] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [10941.627061] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10947.573486] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [10951.281627] Lustre: 279265:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10958.454264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10964.994436] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [10969.025446] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10974.572137] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10980.367991] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10983.110590] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [10992.987960] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10997.925538] Lustre: 281144:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11003.759314] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [11004.027487] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [11016.002727] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 12:03:03 (1789488183) [11034.085334] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11034.099167] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11034.116196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11034.120504] Lustre: Skipped 4 previous similar messages [11047.392661] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11047.679647] Lustre: server umount lustre-MDT0000 complete [11051.612059] LustreError: 278104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789488220 with bad export cookie 12642100438916527431 [11051.624171] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11051.638694] LustreError: 278104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11052.024549] LustreError: 278119:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11052.042621] LustreError: 278119:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [11058.175869] Lustre: server umount lustre-MDT0001 complete [11069.223517] Lustre: server umount lustre-OST0000 complete [11073.352366] Lustre: server umount lustre-OST0001 complete [11081.229101] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [11093.299526] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [11150.636757] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [11163.734039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11164.169728] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11164.195683] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11164.249662] Lustre: lustre-MDT0000: new disk, initializing [11164.309790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11168.714449] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [11180.553291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11180.701983] Lustre: 285388:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11181.119564] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11181.143614] Lustre: Skipped 4 previous similar messages [11181.232509] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11188.051541] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [11204.652290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11205.253674] Lustre: 287020:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11207.222093] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11213.946909] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [11226.198149] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11226.402667] Lustre: 287892:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11228.337638] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11228.370032] Lustre: Skipped 1 previous similar message [11228.534153] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11233.379846] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [11243.551841] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11247.678917] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11247.684268] Lustre: Skipped 1 previous similar message [11266.542654] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 12:07:13 (1789488433) [11269.604857] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11269.612143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11269.624581] Lustre: Skipped 6 previous similar messages [11269.635592] Lustre: Skipped 19 previous similar messages [11271.323589] Lustre: server umount lustre-MDT0000 complete [11275.733214] LustreError: 286183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789488444 with bad export cookie 12642100438916535306 [11275.746320] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11275.755415] LustreError: 286183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11276.269443] Lustre: server umount lustre-MDT0001 complete [11290.921989] Lustre: server umount lustre-OST0000 complete [11305.634959] Lustre: server umount lustre-OST0001 complete [11315.081626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11315.665660] LustreError: 290139:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11315.693602] LustreError: 290139:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [11320.732373] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [11329.769613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11335.587913] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [11344.217643] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11351.831583] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11359.892220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11367.734951] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [11376.575587] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11395.260273] loop1: detected capacity change from 0 to 204800000 [11395.683417] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11395.728777] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11395.757680] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11401.665122] loop1: detected capacity change from 0 to 204800000 [11407.544206] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11407.667230] LustreError: 290126:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs8997: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11407.675482] LustreError: 290126:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs8997 [11407.680231] LustreError: 290126:0:(mgs_handler.c:517:mgs_target_reg()) fs8997-MDT0001: Error checking target index: rc = -13 [11407.687928] LustreError: fs8997-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11407.694426] LustreError: 293860:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [11407.700931] LustreError: 293860:0:(tgt_mount.c:2132:server_put_super()) no obd fs8997-MDT0001 [11407.706328] LustreError: 293860:0:(tgt_mount.c:116:server_deregister_mount()) fs8997-MDT0001 not registered [11407.745682] Lustre: server umount fs8997-MDT0001 complete [11407.749077] LustreError: 293860:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11417.688297] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11417.759315] Lustre: Setting parameter fs8997-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs8997-MDT0001 [11417.803286] Lustre: srv-fs8997-MDT0001: No data found on store. Initialize space. [11417.810384] Lustre: Skipped 4 previous similar messages [11417.871567] Lustre: fs8997-MDT0001: new disk, initializing [11417.876304] Lustre: Skipped 3 previous similar messages [11417.991164] Lustre: srv-fs8997-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11418.001646] Lustre: 294326:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs8997-MDT0001-mdtlov: can't create new llog: rc = -115 [11418.007889] LustreError: 294326:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs8997-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11422.328760] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [11429.991284] Lustre: server umount fs8997-MDT0001 complete [11442.161122] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11446.537489] loop1: detected capacity change from 0 to 204800000 [11453.626922] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11454.061393] Lustre: 295893:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs8997-MDT0001-mdtlov: can't create new llog: rc = -115 [11454.071481] LustreError: 295893:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs8997-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11457.522338] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11458.945830] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [11464.146114] Lustre: server umount fs8997-MDT0001 complete [11477.992313] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11477.996910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11478.016501] Lustre: Skipped 4 previous similar messages [11478.027447] Lustre: Skipped 2 previous similar messages [11481.108080] Lustre: server umount lustre-MDT0000 complete [11490.899848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11491.004575] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11496.437477] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11497.086799] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [11505.070683] loop1: detected capacity change from 0 to 204800000 [11510.569914] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11511.015756] Lustre: 298341:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs8997-MDT0001-mdtlov: can't create new llog: rc = -115 [11511.029114] LustreError: 298341:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs8997-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11515.223199] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [11519.615194] Lustre: server umount fs8997-MDT0001 complete [11542.507414] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11548.449689] Lustre: server umount lustre-OST0000 complete [11559.572140] Lustre: server umount lustre-MDT0000 complete [11563.121747] LustreError: 290120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789488732 with bad export cookie 12642100438916537987 [11563.134621] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11563.363650] Lustre: server umount lustre-MDT0001 complete [11573.744756] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [11576.967416] Key type lgssc unregistered [11577.241742] LNet: 300688:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11577.245850] LNetError: 300688:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11578.281232] LNet: Removed LNI 192.168.202.133@tcp [11579.530284] Key type .llcrypt unregistered [11579.532151] Key type ._llcrypt unregistered [11601.670247] Key type ._llcrypt registered [11601.678314] Key type .llcrypt registered [11601.856614] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [11603.126000] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11603.218394] alg: No test for adler32 (adler32-zlib) [11604.272568] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [11604.518764] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [11606.273746] Key type lgssc registered [11607.298460] Lustre: Echo OBD driver; http://www.lustre.org/ [11615.151613] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11615.189387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11616.625691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11621.812965] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [11626.978851] LustreError: 301961:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11626.995601] LustreError: 301961:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11630.730043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11631.333812] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11636.484955] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [11644.470889] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11650.346520] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11659.072594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11659.755471] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11662.836763] LustreError: 303795:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11667.008587] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [11667.944217] LustreError: 303794:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11673.074932] LustreError: 303793:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11676.683398] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11679.442952] LustreError: 303794:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11679.467812] LustreError: 303794:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11689.961177] LustreError: 303793:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11689.982837] LustreError: 303793:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11697.815667] loop1: detected capacity change from 0 to 204800000 [11698.009481] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11698.028668] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11698.047747] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11704.185394] loop1: detected capacity change from 0 to 204800000 [11706.853653] LustreError: 303793:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11706.888437] LustreError: 303793:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [11710.085898] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11710.251516] Lustre: fs18059-OST0000: new disk, initializing [11710.257588] Lustre: srv-fs18059-OST0000: No data found on store. Initialize space. [11710.263376] Lustre: 305703:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11710.364494] Lustre: fs18059-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11716.604825] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [11725.762774] Lustre: server umount fs18059-OST0000 complete [11741.137493] LustreError: 303793:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11741.161658] loop1: detected capacity change from 0 to 204800000 [11741.189423] LustreError: 303793:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 18 previous similar messages [11742.716295] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11750.557781] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11751.020840] Lustre: fs18059-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11758.077068] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11758.330286] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [11769.469963] loop2: detected capacity change from 0 to 204800000 [11769.886749] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11769.908842] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11769.929246] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11776.043770] loop2: detected capacity change from 0 to 204800000 [11782.745155] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11782.918851] LustreError: fs18059-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11782.943754] LustreError: fs18059-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11782.959774] LustreError: 308744:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [11782.969830] LustreError: 308744:0:(tgt_mount.c:2132:server_put_super()) no obd fs18059-OST0001 [11782.988976] LustreError: 308744:0:(tgt_mount.c:116:server_deregister_mount()) fs18059-OST0001 not registered [11783.061575] Lustre: server umount fs18059-OST0001 complete [11783.063680] LustreError: 308744:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11796.586901] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11796.719948] Lustre: fs18059-OST0001: new disk, initializing [11796.722437] Lustre: srv-fs18059-OST0001: No data found on store. Initialize space. [11796.726746] Lustre: 309295:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11796.777388] Lustre: fs18059-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11804.327614] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [11807.716628] LustreError: 303966:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11807.750072] LustreError: 303966:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [11814.013916] Lustre: server umount fs18059-OST0000 complete [11823.718397] Lustre: server umount fs18059-OST0001 complete [11837.419607] loop1: detected capacity change from 0 to 204800000 [11843.098437] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11843.363162] Lustre: fs18059-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11850.148545] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [11857.588081] loop2: detected capacity change from 0 to 204800000 [11864.359413] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11871.477941] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [11880.598740] Lustre: server umount fs18059-OST0000 complete [11890.318782] Lustre: server umount fs18059-OST0001 complete [11896.310371] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11903.459195] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11903.480155] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11903.519406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11906.534491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11906.559759] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11909.003564] Lustre: server umount lustre-OST0000 complete [11912.678917] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11912.688812] LustreError: Skipped 1 previous similar message [11912.698860] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11912.719366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11912.721867] Lustre: Skipped 1 previous similar message [11916.771804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11916.772471] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11919.275492] Lustre: server umount lustre-MDT0000 complete [11923.598492] LustreError: 301942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789489092 with bad export cookie 17989704997099688784 [11923.606192] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11923.611786] LustreError: 301942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11923.895684] Lustre: server umount lustre-MDT0001 complete [11935.693693] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [11938.676439] Key type lgssc unregistered [11938.980208] LNet: 315221:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11938.993104] LNetError: 315221:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11939.025902] LNet: Removed LNI 192.168.202.133@tcp [11939.918237] Key type .llcrypt unregistered [11939.922217] Key type ._llcrypt unregistered [11977.099972] Key type ._llcrypt registered [11977.104280] Key type .llcrypt registered [11977.261700] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [11994.445972] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [11995.708037] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11995.832166] alg: No test for adler32 (adler32-zlib) [11997.056227] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [11997.541801] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [11999.296171] Key type lgssc registered [12000.442383] Lustre: Echo OBD driver; http://www.lustre.org/ [12039.625763] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12039.676234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12041.017358] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12041.070372] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12041.174167] Lustre: lustre-MDT0000: new disk, initializing [12041.280633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12041.309995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12046.763837] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [12057.156859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12057.315775] Lustre: 319100:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12057.371349] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12057.376232] Lustre: Skipped 1 previous similar message [12057.427449] Lustre: lustre-MDT0001: new disk, initializing [12057.502876] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12057.539664] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12057.550705] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12062.459769] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [12071.905686] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12077.672527] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12084.236557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12084.559983] Lustre: lustre-OST0000: new disk, initializing [12084.568579] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12084.582104] Lustre: 321071:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12084.721537] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12090.806713] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [12094.551509] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12094.572566] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12094.681489] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12100.764854] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12102.785241] Lustre: Mounted lustre-client - version 2.17.57_84_gd41d3c6 [12103.892194] Lustre: Unmounted lustre-client [12115.436394] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12115.439965] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12115.448095] LustreError: Skipped 1 previous similar message [12115.467449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12115.478354] Lustre: Skipped 1 previous similar message [12118.501629] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12118.505125] Lustre: Skipped 1 previous similar message [12119.938369] Lustre: server umount lustre-OST0000 complete [12128.747244] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12128.766854] Lustre: Skipped 1 previous similar message [12128.778414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12128.786649] Lustre: Skipped 1 previous similar message [12130.821237] Lustre: server umount lustre-MDT0000 complete [12134.836164] LustreError: 319897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789489303 with bad export cookie 4793266440963155379 [12134.837314] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12134.842293] LustreError: 319897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12135.086513] Lustre: server umount lustre-MDT0001 complete [12146.044938] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [12148.824801] Key type lgssc unregistered [12153.187803] LNet: 1 local NIs in recovery (showing 1): 192.168.202.133@tcp [12153.255951] LNet: 323234:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12153.284810] LNetError: 323234:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12232.166656] LNet: Removed LNI 192.168.202.133@tcp [12232.988959] Key type .llcrypt unregistered [12232.991831] Key type ._llcrypt unregistered [12253.862487] Key type ._llcrypt registered [12253.865450] Key type .llcrypt registered [12254.011926] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [12271.327517] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [12272.609322] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12272.859564] alg: No test for adler32 (adler32-zlib) [12273.959738] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [12274.261363] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [12275.952411] Key type lgssc registered [12277.134247] Lustre: Echo OBD driver; http://www.lustre.org/ [12309.016822] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 12:24:36 (1789489476) [12316.637138] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12316.726049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12318.052480] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12318.088240] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12318.145495] Lustre: lustre-MDT0000: new disk, initializing [12318.222927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12318.233841] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12323.507077] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [12335.202058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12335.339840] Lustre: 326460:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12335.447151] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12335.453458] Lustre: Skipped 1 previous similar message [12335.514210] Lustre: lustre-MDT0001: new disk, initializing [12335.593853] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12335.632924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12335.643150] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12340.921779] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [12351.386488] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12357.926635] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12365.818171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12366.169820] Lustre: lustre-OST0000: new disk, initializing [12366.175328] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12366.188513] Lustre: 328431:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12366.360440] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12373.543890] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [12374.645424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12374.649806] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12374.693558] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12385.200721] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12405.734915] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12405.737501] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12405.744168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12405.755772] LustreError: Skipped 1 previous similar message [12405.796350] Lustre: Skipped 1 previous similar message [12410.853089] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12410.860705] Lustre: Skipped 1 previous similar message [12411.824123] Lustre: server umount lustre-OST0000 complete [12417.505519] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12417.511278] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12417.525202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12421.090637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12421.094382] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12426.220159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12426.223388] Lustre: Skipped 2 previous similar messages [12431.331862] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12431.628562] Lustre: server umount lustre-MDT0000 complete [12435.285121] LustreError: 326452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789489604 with bad export cookie 3234288502361300605 [12435.292628] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12435.302534] LustreError: 326452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12435.669976] Lustre: server umount lustre-MDT0001 complete [12447.759271] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [12450.745089] Key type lgssc unregistered [12451.163724] LNet: 330344:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12451.177789] LNetError: 330344:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12451.195917] LNet: Removed LNI 192.168.202.133@tcp [12452.064358] Key type .llcrypt unregistered [12452.066737] Key type ._llcrypt unregistered [12475.301235] Key type ._llcrypt registered [12475.309797] Key type .llcrypt registered [12475.413584] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [12476.677569] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12476.905122] alg: No test for adler32 (adler32-zlib) [12478.117466] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [12478.403380] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [12480.184235] Key type lgssc registered [12481.545599] Lustre: Echo OBD driver; http://www.lustre.org/ [12489.600233] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12489.614914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12490.970900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12495.570959] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [12500.523374] LustreError: 331620:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12500.562184] LustreError: 331620:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12505.573130] LustreError: 331619:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12507.027698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12507.287596] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12512.870443] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [12522.435405] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12529.259666] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12538.342216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12538.741680] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12545.495147] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [12556.466192] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12572.259816] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12574.323235] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12580.839609] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12580.846299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12580.862076] Lustre: Skipped 2 previous similar messages [12580.882853] Lustre: Skipped 2 previous similar messages [12582.910090] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.33@tcp (stopping) [12585.872749] Lustre: server umount lustre-MDT0000 complete [12585.959115] LustreError: 331614:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12585.995843] LustreError: 331614:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12587.985973] LustreError: 332908:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12590.479207] LustreError: 331599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789489759 with bad export cookie 13752177028978907380 [12590.490772] LustreError: 331599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12590.492959] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12591.073660] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12591.082910] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12593.101658] LustreError: 332908:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12593.123905] LustreError: 332908:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12593.144405] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.33@tcp (stopping) [12596.936211] Lustre: server umount lustre-MDT0001 complete [12601.819178] Lustre: server umount lustre-OST0000 complete [12612.158309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12618.351307] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [12621.469533] Lustre: 335807:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.133@tcp [12623.758132] Lustre: server umount MGS complete [12632.214595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12632.635158] LustreError: 336378:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12632.648617] LustreError: 336378:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12632.704583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12637.149907] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [12648.519340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12649.172941] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12655.703889] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [12665.658620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12666.043264] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12673.572782] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [12683.203326] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12685.219544] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12691.866396] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12693.448720] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12699.134829] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [12699.141976] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12699.165769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12699.167938] Lustre: Skipped 1 previous similar message [12700.139096] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12700.884370] Lustre: server umount lustre-OST0000 complete [12705.257456] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12705.270223] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12705.281464] Lustre: Skipped 1 previous similar message [12708.322097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12708.335516] Lustre: Skipped 2 previous similar messages [12711.709529] Lustre: server umount lustre-MDT0000 complete [12713.448993] LustreError: 336375:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12713.480211] LustreError: 336375:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [12716.895601] LustreError: 336358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789489885 with bad export cookie 13752177028978908220 [12716.898696] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12716.903751] LustreError: 336358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12717.482636] Lustre: server umount lustre-MDT0001 complete [12732.285643] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [12735.586503] Key type lgssc unregistered [12735.894556] LNet: 339820:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12735.906031] LNetError: 339820:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12735.919177] LNet: Removed LNI 192.168.202.133@tcp [12736.982694] Key type .llcrypt unregistered [12736.986338] Key type ._llcrypt unregistered [12762.250435] Key type ._llcrypt registered [12762.252421] Key type .llcrypt registered [12762.405880] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [12781.736977] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [12783.249912] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12783.275219] alg: No test for adler32 (adler32-zlib) [12784.813828] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [12785.216078] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [12787.000231] Key type lgssc registered [12788.497963] Lustre: Echo OBD driver; http://www.lustre.org/ [12833.674303] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12833.721338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12835.134658] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12835.221773] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12835.383856] Lustre: lustre-MDT0000: new disk, initializing [12835.494137] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12835.520862] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12840.479562] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [12854.013523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12854.264849] Lustre: 342990:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12854.414746] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12854.428561] Lustre: Skipped 1 previous similar message [12854.565000] Lustre: lustre-MDT0001: new disk, initializing [12854.753089] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12854.815059] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12854.828284] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12862.420574] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [12872.827418] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12880.223722] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12888.767746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12889.290718] Lustre: lustre-OST0000: new disk, initializing [12889.298959] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12889.311878] Lustre: 344958:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12889.455131] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12896.382538] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12896.394366] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12896.529138] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12897.398706] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [12907.625640] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12913.039886] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12913.262917] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12921.575125] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12921.951352] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12927.460167] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12927.466242] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12927.479831] LustreError: Skipped 1 previous similar message [12927.480835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12927.521227] Lustre: Skipped 1 previous similar message [12931.497363] Lustre: server umount lustre-OST0000 complete [12936.673589] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12936.676824] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12936.693569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12936.702335] Lustre: Skipped 1 previous similar message [12937.704694] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12937.705949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12942.512787] Lustre: server umount lustre-MDT0000 complete [12947.557968] LustreError: 342982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789490116 with bad export cookie 6213699796293637039 [12947.567219] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12947.573590] LustreError: 342982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12947.824262] Lustre: server umount lustre-MDT0001 complete [12958.981654] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 12:35:25 (1789490125) [12980.359535] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [12992.635820] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [13037.913102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13038.303341] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13038.323271] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13038.385628] Lustre: lustre-MDT0000: new disk, initializing [13038.457067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13038.488908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13044.047490] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [13055.814667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13056.020434] Lustre: 349725:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13056.081187] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13056.088509] Lustre: Skipped 1 previous similar message [13056.193978] Lustre: lustre-MDT0001: new disk, initializing [13056.309851] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13056.381577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13056.404719] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13062.371927] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [13076.464529] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13084.501610] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13093.045877] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13093.304360] Lustre: lustre-OST0000: new disk, initializing [13093.311332] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13093.323172] Lustre: 351694:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13093.456768] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13101.201631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13101.218203] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13101.355189] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13101.865255] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [13113.190460] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13157.868195] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13157.883466] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13157.898523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13157.904379] Lustre: Skipped 2 previous similar messages [13159.466431] Lustre: server umount lustre-OST0000 complete [13163.534849] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [13163.547176] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13163.561095] Lustre: Skipped 1 previous similar message [13163.571081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13170.035985] Lustre: server umount lustre-MDT0000 complete [13173.223970] LustreError: 349729:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13173.276982] LustreError: 349729:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13174.650991] LustreError: 349715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789490343 with bad export cookie 6213699796293637921 [13174.662697] LustreError: 349715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13174.670303] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13175.136754] Lustre: server umount lustre-MDT0001 complete [13189.039988] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [13192.805352] Key type lgssc unregistered [13193.216087] LNet: 353889:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13193.237835] LNetError: 353889:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13193.263757] LNet: Removed LNI 192.168.202.133@tcp [13194.647698] Key type .llcrypt unregistered [13194.650845] Key type ._llcrypt unregistered [13305.701408] Key type ._llcrypt registered [13305.703492] Key type .llcrypt registered [13305.807820] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [13306.116319] Key type .llcrypt unregistered [13306.120543] Key type ._llcrypt unregistered [13394.936246] Key type ._llcrypt registered [13394.937683] Key type .llcrypt registered [13395.026232] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [13395.392685] Key type .llcrypt unregistered [13395.394300] Key type ._llcrypt unregistered [13427.283299] Key type ._llcrypt registered [13427.285905] Key type .llcrypt registered [13427.384117] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [13427.655305] Key type .llcrypt unregistered [13427.657146] Key type ._llcrypt unregistered [13464.186268] Key type ._llcrypt registered [13464.188808] Key type .llcrypt registered [13464.281525] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [13464.614351] Key type .llcrypt unregistered [13464.616524] Key type ._llcrypt unregistered [13559.948423] Key type ._llcrypt registered [13559.959746] Key type .llcrypt registered [13560.153966] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [13560.824322] Key type .llcrypt unregistered [13560.826386] Key type ._llcrypt unregistered [13602.752355] Key type ._llcrypt registered [13602.755972] Key type .llcrypt registered [13602.955412] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [13603.606517] Key type .llcrypt unregistered [13603.610076] Key type ._llcrypt unregistered [13638.981351] Key type ._llcrypt registered [13638.983053] Key type .llcrypt registered [13639.152322] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [13639.394472] Key type .llcrypt unregistered [13639.395516] Key type ._llcrypt unregistered [13673.563612] Key type ._llcrypt registered [13673.569347] Key type .llcrypt registered [13673.803546] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [13674.347579] Key type .llcrypt unregistered [13674.354366] Key type ._llcrypt unregistered [13713.819279] Key type ._llcrypt registered [13713.820900] Key type .llcrypt registered [13713.936739] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [13714.315515] Key type .llcrypt unregistered [13714.317075] Key type ._llcrypt unregistered [13751.412861] Key type ._llcrypt registered [13751.414306] Key type .llcrypt registered [13751.560601] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [13751.975408] Key type .llcrypt unregistered [13751.981728] Key type ._llcrypt unregistered [13833.103935] Key type ._llcrypt registered [13833.108766] Key type .llcrypt registered [13833.235539] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [13833.599449] Key type .llcrypt unregistered [13833.605194] Key type ._llcrypt unregistered [13873.012722] Key type ._llcrypt registered [13873.014372] Key type .llcrypt registered [13873.112253] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [13873.502629] Key type .llcrypt unregistered [13873.504614] Key type ._llcrypt unregistered [13910.582763] Key type ._llcrypt registered [13910.584894] Key type .llcrypt registered [13910.709927] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [13911.057554] Key type .llcrypt unregistered [13911.060430] Key type ._llcrypt unregistered [13968.535626] Key type ._llcrypt registered [13968.537761] Key type .llcrypt registered [13968.651083] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [13987.539854] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [13988.889623] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13988.991268] alg: No test for adler32 (adler32-zlib) [13990.235617] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [13990.433482] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [13992.096640] Key type lgssc registered [13993.426174] Lustre: Echo OBD driver; http://www.lustre.org/ [14036.983145] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14037.047737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14038.423977] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [14038.504482] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [14038.681983] Lustre: tst250fs-MDT0000: new disk, initializing [14038.851880] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14038.884752] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14045.112670] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [14059.174118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14059.318780] Lustre: 374324:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14059.414145] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [14059.417134] Lustre: Skipped 1 previous similar message [14059.518824] Lustre: tst250fs-MDT0001: new disk, initializing [14059.670195] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14059.736372] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14059.764350] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14065.798939] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [14071.833758] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [14079.733415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14080.313858] Lustre: tst250fs-OST0000: new disk, initializing [14080.322911] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [14080.330851] Lustre: 375957:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14080.546073] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14087.420992] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [14088.822434] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14088.832700] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14088.953685] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14099.380026] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14099.655854] Lustre: tst250fs-OST0001: new disk, initializing [14099.658895] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [14099.666750] Lustre: 376829:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14099.793209] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [14105.196563] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [14105.210913] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [14105.349229] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [14107.098479] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [14113.556822] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [14115.572857] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [14115.582447] Lustre: Skipped 1 previous similar message [14136.291154] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14136.292400] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14136.294575] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14136.323484] Lustre: Skipped 1 previous similar message [14141.410501] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14141.418079] Lustre: Skipped 3 previous similar messages [14141.945119] Lustre: server umount tst250fs-MDT0000 complete [14146.637266] LustreError: 375120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789491315 with bad export cookie 10710169241388950539 [14146.643798] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14146.667905] LustreError: 375120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14147.292320] Lustre: server umount tst250fs-MDT0001 complete [14162.419504] Lustre: server umount tst250fs-OST0000 complete [14176.923469] Lustre: server umount tst250fs-OST0001 complete [14200.709730] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [14211.123941] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [14245.420474] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 12:56:52 (1789491412) [14248.020417] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [14267.295869] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [14280.055288] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [14314.551662] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14001 sec ========= 12:58:01 (1789491481) [14317.462255] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:58:03 (1789491483) === [14323.320575] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:58:09 (1789491489) === [14376.551868] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [14379.588783] Key type lgssc unregistered [14379.856092] LNet: 385977:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14379.860458] LNetError: 385977:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14379.877531] LNet: Removed LNI 192.168.202.133@tcp [14380.866177] Key type .llcrypt unregistered [14380.872869] Key type ._llcrypt unregistered