[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 471999137 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.003304] x2apic enabled [ 0.004011] Switched APIC routing to physical x2apic. [ 0.005018] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008029] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009021] pid_max: default: 32768 minimum: 301 [ 0.010170] LSM: Security Framework initializing [ 0.011075] Yama: becoming mindful. [ 0.013035] SELinux: Initializing. [ 0.014081] *** VALIDATE selinux *** [ 0.023039] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028653] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030021] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032098] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033143] *** VALIDATE tmpfs *** [ 0.035533] *** VALIDATE proc *** [ 0.036319] *** VALIDATE cgroup *** [ 0.037012] *** VALIDATE cgroup2 *** [ 0.038230] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039157] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041038] Spectre V2 : User space: Vulnerable [ 0.042013] Speculative Store Bypass: Vulnerable [ 0.045633] debug: unmapping init [mem 0xffffffffa3e59000-0xffffffffa3e60fff] [ 0.047954] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048747] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049033] ... version: 2 [ 0.050020] ... bit width: 48 [ 0.051015] ... generic registers: 4 [ 0.052015] ... value mask: 0000ffffffffffff [ 0.053021] ... max period: 00007fffffffffff [ 0.054020] ... fixed-purpose events: 3 [ 0.055016] ... event mask: 000000070000000f [ 0.057297] rcu: Hierarchical SRCU implementation. [ 0.059763] smp: Bringing up secondary CPUs ... [ 0.060666] x86: Booting SMP configuration: [ 0.061032] .... node #0, CPUs: #1 #2 #3 [ 0.069244] smp: Brought up 1 node, 4 CPUs [ 0.071019] smpboot: Max logical packages: 1 [ 0.072027] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.196656] node 0 deferred pages initialised in 121ms [ 0.198142] devtmpfs: initialized [ 0.199221] x86/mm: Memory block size: 128MB [ 0.201922] gcov: version magic: 0x41383552 [ 0.205309] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.206129] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.207348] pinctrl core: initialized pinctrl subsystem [ 0.208243] [ 0.208734] ************************************************************* [ 0.209021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.210021] ** ** [ 0.211020] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.212018] ** ** [ 0.213019] ** This means that this kernel is built to expose internal ** [ 0.214018] ** IOMMU data structures, which may compromise security on ** [ 0.215019] ** your system. ** [ 0.216016] ** ** [ 0.217018] ** If you see this message and you are not debugging the ** [ 0.218016] ** kernel, report this immediately to your vendor! ** [ 0.219018] ** ** [ 0.220017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.221016] ************************************************************* [ 0.222723] NET: Registered protocol family 16 [ 0.223406] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.224049] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.225066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.226599] cpuidle: using governor menu [ 0.227767] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.229496] PCI: Using configuration type 1 for base access [ 0.231105] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.237120] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.238043] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.241080] cryptd: max_cpu_qlen set to 1000 [ 0.243255] ACPI: Added _OSI(Module Device) [ 0.244015] ACPI: Added _OSI(Processor Device) [ 0.245015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.246011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.250187] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.256815] ACPI: Interpreter enabled [ 0.257069] ACPI: PM: (supports S0 S3 S4 S5) [ 0.258017] ACPI: Using IOAPIC for interrupt routing [ 0.259137] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.260490] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.269729] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.270067] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.271029] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.272103] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.274474] acpiphp: Slot [2] registered [ 0.275199] acpiphp: Slot [5] registered [ 0.276215] acpiphp: Slot [6] registered [ 0.277173] acpiphp: Slot [7] registered [ 0.278172] acpiphp: Slot [8] registered [ 0.279153] acpiphp: Slot [9] registered [ 0.280171] acpiphp: Slot [10] registered [ 0.281151] acpiphp: Slot [3] registered [ 0.282098] acpiphp: Slot [4] registered [ 0.283100] acpiphp: Slot [11] registered [ 0.284150] acpiphp: Slot [12] registered [ 0.285108] acpiphp: Slot [13] registered [ 0.286138] acpiphp: Slot [14] registered [ 0.287162] acpiphp: Slot [15] registered [ 0.289149] acpiphp: Slot [16] registered [ 0.291142] acpiphp: Slot [17] registered [ 0.293135] acpiphp: Slot [18] registered [ 0.295140] acpiphp: Slot [19] registered [ 0.296095] acpiphp: Slot [20] registered [ 0.297111] acpiphp: Slot [21] registered [ 0.298087] acpiphp: Slot [22] registered [ 0.299063] acpiphp: Slot [23] registered [ 0.300131] acpiphp: Slot [24] registered [ 0.302126] acpiphp: Slot [25] registered [ 0.303151] acpiphp: Slot [26] registered [ 0.305175] acpiphp: Slot [27] registered [ 0.307170] acpiphp: Slot [28] registered [ 0.309131] acpiphp: Slot [29] registered [ 0.310142] acpiphp: Slot [30] registered [ 0.312148] acpiphp: Slot [31] registered [ 0.314117] PCI host bridge to bus 0000:00 [ 0.316033] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.318033] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.321038] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.324036] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.326032] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.329046] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.331303] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.334476] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.337574] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.347799] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.352508] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.355021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.357018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.359019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.362237] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.364907] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.384023] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.387102] pci 0000:00:01.3: quirk_piix4_acpi+0x0/0x1e0 took 21484 usecs [ 0.391019] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.397019] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.410018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.416021] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.423201] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.431021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.440027] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.462017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.474794] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.487024] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.505026] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.523023] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.537254] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.545017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.552023] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.571039] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.581898] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.589025] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.597021] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.619022] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.633390] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.643021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.653020] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.668024] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.678720] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.685022] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.691025] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.720021] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.735176] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.737444] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.740395] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.742457] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.744228] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.748191] iommu: Default domain type: Passthrough [ 0.750444] SCSI subsystem initialized [ 0.752156] ACPI: bus type USB registered [ 0.753104] usbcore: registered new interface driver usbfs [ 0.755083] usbcore: registered new interface driver hub [ 0.756062] usbcore: registered new device driver usb [ 0.758232] pps_core: LinuxPPS API ver. 1 registered [ 0.760101] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.763087] PTP clock support registered [ 0.765083] EDAC MC: Ver: 3.0.0 [ 0.767232] PCI: Using ACPI for IRQ routing [ 0.768968] NetLabel: Initializing [ 0.770018] NetLabel: domain hash size = 128 [ 0.772016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.775110] NetLabel: unlabeled traffic allowed by default [ 0.778100] vgaarb: loaded [ 0.779297] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.780016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.787000] clocksource: Switched to clocksource kvm-clock [ 0.903044] VFS: Disk quotas dquot_6.6.0 [ 0.905725] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.909338] *** VALIDATE ramfs *** [ 0.910956] *** VALIDATE hugetlbfs *** [ 0.912815] pnp: PnP ACPI init [ 0.916172] pnp: PnP ACPI: found 6 devices [ 0.933387] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.935422] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.937114] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.938308] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.940096] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.942336] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.944421] NET: Registered protocol family 2 [ 0.946425] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.951472] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.956125] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.963144] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.966792] TCP: Hash tables configured (established 65536 bind 65536) [ 0.970246] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.974478] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.978136] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.981503] NET: Registered protocol family 1 [ 0.984989] RPC: Registered named UNIX socket transport module. [ 0.986812] RPC: Registered udp transport module. [ 0.988040] RPC: Registered tcp transport module. [ 0.989180] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.990809] NET: Registered protocol family 44 [ 0.991995] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.993512] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.995209] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.996806] PCI: CLS 0 bytes, default 64 [ 0.998238] Unpacking initramfs... [ 2.375178] debug: unmapping init [mem 0xffff8d07bcc54000-0xffff8d07bffbffff] [ 2.379261] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.381704] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.384802] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.866314] Initialise system trusted keyrings [ 2.867825] Key type blacklist registered [ 2.869899] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.879252] zbud: loaded [ 2.882262] *** VALIDATE nfs *** [ 2.883449] *** VALIDATE nfs4 *** [ 2.884954] pstore: using deflate compression [ 2.888847] Platform Keyring initialized [ 2.996796] NET: Registered protocol family 38 [ 2.998277] Key type asymmetric registered [ 2.999417] Asymmetric key parser 'x509' registered [ 3.000761] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.003032] io scheduler mq-deadline registered [ 3.004434] io scheduler kyber registered [ 3.005943] io scheduler bfq registered [ 3.008614] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.011154] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.013489] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.015536] ACPI: Power Button [PWRF] [ 3.020510] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.027673] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.055078] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.074437] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.092312] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.120535] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.157537] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.162504] Non-volatile memory driver v1.3 [ 3.164156] Linux agpgart interface v0.103 [ 3.196487] virtio_blk virtio1: [vda] 146184 512-byte logical blocks (74.8 MB/71.4 MiB) [ 3.199462] vda: detected capacity change from 0 to 74846208 [ 3.213632] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.216400] vdb: detected capacity change from 0 to 1073741824 [ 3.232476] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.237636] vdc: detected capacity change from 0 to 2621440000 [ 3.252890] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.256459] vdd: detected capacity change from 0 to 2621440000 [ 3.278625] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.282107] vde: detected capacity change from 0 to 4294967296 [ 3.295816] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.298309] vdf: detected capacity change from 0 to 4294967296 [ 3.303847] libphy: Fixed MDIO Bus: probed [ 3.312454] usbcore: registered new interface driver usbserial_generic [ 3.314936] usbserial: USB Serial support registered for generic [ 3.317156] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.321405] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.322932] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.325504] mousedev: PS/2 mouse device common for all mice [ 3.328493] rtc_cmos 00:05: RTC can wake from S4 [ 3.335758] rtc_cmos 00:05: registered as rtc0 [ 3.336266] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.337548] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.343540] intel_pstate: CPU model not supported [ 3.346512] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.350538] hid: raw HID events driver (C) Jiri Kosina [ 3.353235] usbcore: registered new interface driver usbhid [ 3.355348] usbhid: USB HID core driver [ 3.356901] drop_monitor: Initializing network drop monitor service [ 3.357220] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.359194] Initializing XFRM netlink socket [ 3.364287] NET: Registered protocol family 10 [ 3.367399] Segment Routing with IPv6 [ 3.368485] NET: Registered protocol family 17 [ 3.370408] mpls_gso: MPLS GSO support [ 3.376274] RAS: Correctable Errors collector initialized. [ 3.377739] AVX version of gcm_enc/dec engaged. [ 3.378798] AES CTR mode by8 optimization enabled [ 3.457926] sched_clock: Marking stable (3457899548, 0)->(4410217313, -952317765) [ 3.461846] registered taskstats version 1 [ 3.464339] Loading compiled-in X.509 certificates [ 3.466797] zswap: loaded using pool lzo/zbud [ 3.491616] Key type big_key registered [ 3.514599] Key type encrypted registered [ 3.516062] ima: No TPM chip found, activating TPM-bypass! [ 3.518023] ima: Allocated hash algorithm: sha1 [ 3.519776] ima: No architecture policies found [ 3.521643] evm: Initialising EVM extended attributes: [ 3.523463] evm: security.selinux [ 3.524608] evm: security.ima [ 3.525514] evm: security.capability [ 3.526793] evm: HMAC attrs: 0x1 [ 3.529597] rtc_cmos 00:05: setting system clock to 2026-08-19 10:12:53 UTC (1787134373) [ 3.535846] debug: unmapping init [mem 0xffffffffa4e03000-0xffffffffa4ffffff] [ 3.539379] debug: unmapping init [mem 0xffffffffa3b82000-0xffffffffa3e58fff] [ 3.552145] Write protecting the kernel read-only data: 28672k [ 3.555597] debug: unmapping init [mem 0xffffffffa2203000-0xffffffffa23fffff] [ 3.558495] debug: unmapping init [mem 0xffffffffa2b14000-0xffffffffa2bfffff] [ 3.594826] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.603145] systemd[1]: Detected virtualization kvm. [ 3.604877] systemd[1]: Detected architecture x86-64. [ 3.606652] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.635706] systemd[1]: No hostname configured. [ 3.638631] systemd[1]: Set hostname to . [ 3.640953] random: systemd: uninitialized urandom read (16 bytes read) [ 3.642874] systemd[1]: Initializing machine ID from random generator. [ 3.793697] random: systemd: uninitialized urandom read (16 bytes read) [ 3.797323] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.801928] random: systemd: uninitialized urandom read (16 bytes read) [ 3.804914] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.809976] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. [ OK ] Reached target Initrd Root Device. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Sockets. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.342502] device-mapper: uevent: version 1.0.3 [ 4.344941] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.014564] virtio_net virtio0 ens2: renamed from eth0 [ 5.057495] random: fast init done [ 5.112801] scsi host0: ata_piix [ 5.218585] scsi host1: ata_piix [ 5.220225] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.222523] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.771851] dracut-initqueue[578]: RTNETLINK answers: File exists [ 10.040271] random: crng init done [ 10.041960] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.335875] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.407055] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.632346] SELinux: Disabled at runtime. [ 11.693147] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.716512] systemd[1]: Detected virtualization kvm. [ 11.717857] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.194216] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.197476] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.201617] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.204900] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.208115] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.216188] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.221411] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target rpc_pipefs.target. Mounting Huge Pages File System... [ OK ] Stopped target Initrd File Systems. Mounting Kernel Debug File System... [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on Process Core Dump Socket. Starting Apply Kernel Variables... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-getty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [FAILED] Failed to set up automount Arbitrar…rmat[ 12.367745] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS s File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 12.687237] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.942727] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.052342] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.141102] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.156850] EDAC sbridge: Ver: 1.1.2 [ 14.625394] Key type dns_resolver registered [ 14.926365] NFS: Registering the id_resolver key type [ 14.928444] Key type id_resolver registered [ 14.931259] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg158-server login: [ 31.395034] hrtimer: interrupt took 8016901 ns [ 80.751364] libcfs: loading out-of-tree module taints kernel. [ 80.811057] Key type ._llcrypt registered [ 80.813663] Key type .llcrypt registered [ 80.901046] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 101.747846] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 103.421596] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 103.463926] alg: No test for adler32 (adler32-zlib) [ 104.774226] Lustre: Lustre: Build Version: 2.17.57_11_g9d7c9d8 [ 105.344889] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 107.047176] Key type lgssc registered [ 109.026996] Lustre: Echo OBD driver; http://www.lustre.org/ [ 136.179273] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 181.048153] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 195.191320] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 195.291365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 196.708038] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 196.757851] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 196.864039] Lustre: lustre-MDT0000: new disk, initializing [ 197.023447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 197.073489] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 201.907550] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 215.544211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 215.666985] Lustre: 6509:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 215.704828] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 215.710558] Lustre: Skipped 1 previous similar message [ 215.783764] Lustre: lustre-MDT0001: new disk, initializing [ 215.837840] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 215.861875] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 215.866924] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 220.266416] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 225.381287] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 234.773135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 235.240483] Lustre: lustre-OST0000: new disk, initializing [ 235.249809] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 235.266341] Lustre: 8450:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 235.381176] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 235.846594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 235.864165] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 235.987792] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 242.064241] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 258.759510] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 258.952274] Lustre: lustre-OST0001: new disk, initializing [ 258.957265] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 258.965410] Lustre: 9522:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 259.053955] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 265.354706] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 265.837481] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 265.844233] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 265.903088] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 278.101335] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 286.729952] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 293.674354] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing check_logdir /tmp/testlogs/ [ 299.942906] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing yml_node [ 305.204403] Lustre: DEBUG MARKER: Client: 2.17.57.11 [ 309.046988] Lustre: DEBUG MARKER: MDS: 2.17.57.11 [ 311.864688] Lustre: DEBUG MARKER: OSS: 2.17.57.11 [ 313.769750] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Aug 19 06:18:02 EDT 2026 [ 336.001872] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 338.321881] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 358.367961] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 358.370672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 358.380940] Lustre: Skipped 1 previous similar message [ 358.411216] Lustre: Skipped 3 previous similar messages [ 363.455542] Lustre: server umount lustre-MDT0000 complete [ 367.188185] LustreError: 9521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787134737 with bad export cookie 7668546084553882970 [ 367.191576] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 367.199507] LustreError: 9521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 368.608615] LustreError: 6519:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 368.609368] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 368.624128] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 368.632450] LustreError: 6519:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 6 previous similar messages [ 368.648167] Lustre: Skipped 3 previous similar messages [ 372.640566] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 372.641549] LustreError: 6518:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 372.647330] Lustre: Skipped 1 previous similar message [ 372.691933] LustreError: 6518:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 1 previous similar message [ 373.739205] Lustre: server umount lustre-MDT0001 complete [ 378.399263] Lustre: server umount lustre-OST0000 complete [ 382.941412] Lustre: server umount lustre-OST0001 complete [ 388.783860] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 399.692454] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 436.301566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 436.572269] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 436.607268] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 436.699620] Lustre: lustre-MDT0000: new disk, initializing [ 436.794308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 436.829256] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 442.166738] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 451.875511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 452.014119] Lustre: 15075:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 452.068319] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 452.073485] Lustre: Skipped 1 previous similar message [ 452.132814] Lustre: lustre-MDT0001: new disk, initializing [ 452.231486] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 452.273413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 452.295179] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 456.835221] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 467.369243] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 473.837565] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 482.464266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 483.089080] Lustre: lustre-OST0000: new disk, initializing [ 483.094890] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 483.103511] Lustre: 17042:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 483.314964] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 485.068534] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 485.095914] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 485.354832] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 492.602563] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 504.425821] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 510.131465] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 510.497537] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 515.491413] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 515.684951] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 521.200663] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 521.205727] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 521.222621] LustreError: Skipped 1 previous similar message [ 521.256607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 521.262576] Lustre: Skipped 2 previous similar messages [ 524.613369] Lustre: server umount lustre-OST0000 complete [ 531.425476] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 531.446076] Lustre: Skipped 1 previous similar message [ 531.450812] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 531.461021] Lustre: Skipped 1 previous similar message [ 541.673166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 541.682508] Lustre: Skipped 3 previous similar messages [ 544.223151] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 544.482133] Lustre: server umount lustre-MDT0000 complete [ 546.785120] LustreError: 15883:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 546.794469] LustreError: 15883:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 1 previous similar message [ 548.459376] LustreError: 17052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787134918 with bad export cookie 7668546084553884503 [ 548.466376] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 548.473782] LustreError: 17052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 548.840715] Lustre: server umount lustre-MDT0001 complete [ 555.469559] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 06:22:04 (1787134924) [ 573.247600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 573.692488] LustreError: 19919:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 573.787831] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 577.999353] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 579.041312] LustreError: 19920:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 586.827182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 591.922164] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 599.684202] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 606.031944] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 608.146221] Lustre: Failing over lustre-MDT0000 [ 608.507208] Lustre: server umount lustre-MDT0000 complete [ 608.743388] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 608.764024] Lustre: Skipped 1 previous similar message [ 608.774304] LustreError: 19919:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 608.799568] LustreError: 19919:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 1 previous similar message [ 628.021756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 628.192373] Lustre: 3643:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787134982/real 1787134982] req@ffff8d070794c000 x1873946319427072/t0(0) o400->MGC192.168.201.158@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787134998 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 628.200122] LustreError: 19920:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 628.225909] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 628.300342] LustreError: 19920:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 8 previous similar messages [ 628.684821] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 632.684478] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 632.688850] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 632.722861] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 633.137934] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 642.460315] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 648.163568] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 648.178608] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 648.195652] Lustre: Skipped 1 previous similar message [ 648.200948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 652.670094] Lustre: server umount lustre-MDT0000 complete [ 656.257942] LustreError: 19901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787135026 with bad export cookie 7668546084553885917 [ 656.269215] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 656.274085] LustreError: 19901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 656.656261] Lustre: server umount lustre-MDT0001 complete [ 665.065495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 665.547508] LustreError: 23256:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 665.572559] LustreError: 23256:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 2 previous similar messages [ 665.660155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 665.663068] Lustre: Skipped 2 previous similar messages [ 670.085953] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 679.117842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 684.075308] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 692.106933] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 698.275422] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 700.687995] Lustre: Failing over lustre-MDT0000 [ 700.941662] Lustre: server umount lustre-MDT0000 complete [ 704.808173] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 704.832025] Lustre: Skipped 1 previous similar message [ 720.357061] Lustre: 3641:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787135074/real 1787135074] req@ffff8d070660b800 x1873946319455872/t0(0) o400->MGC192.168.201.158@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787135090 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 720.401176] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 723.307291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 730.592755] LustreError: 23257:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 730.617571] LustreError: 23257:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 15 previous similar messages [ 731.619022] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x6a6c2608c18c0c87 [ 731.645482] Lustre: MGC192.168.201.158@tcp: Connection restored to 0@lo (at 0@lo) [ 731.648015] Lustre: Skipped 1 previous similar message [ 732.124008] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 737.255737] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 737.282556] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 737.341860] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 737.974895] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 748.873564] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 754.657083] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 754.669874] Lustre: Skipped 1 previous similar message [ 754.678733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 754.686441] Lustre: Skipped 3 previous similar messages [ 759.327249] Lustre: server umount lustre-MDT0000 complete [ 763.269129] LustreError: 23238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787135133 with bad export cookie 7668546084553886855 [ 763.273698] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 763.277513] LustreError: 23238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 763.785716] Lustre: server umount lustre-MDT0001 complete [ 773.242917] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 06:25:41 (1787135141) [ 795.124652] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 807.424975] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 848.509987] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 860.081671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 860.421298] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 860.461335] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 860.553860] Lustre: lustre-MDT0000: new disk, initializing [ 860.623948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 860.629887] Lustre: Skipped 2 previous similar messages [ 860.643112] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 865.186966] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 876.285423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 876.493755] Lustre: 29642:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 876.809593] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 881.939651] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 892.741149] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 898.560420] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 906.586960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 906.823808] Lustre: 31616:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 908.137435] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 908.203458] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 908.279471] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 908.301936] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 913.187969] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 923.179903] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 929.220387] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 954.348197] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 954.356121] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 954.357102] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 954.367971] LustreError: Skipped 1 previous similar message [ 957.659799] Lustre: server umount lustre-OST0000 complete [ 964.065494] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 968.825290] Lustre: server umount lustre-MDT0000 complete [ 969.702085] LustreError: 29656:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 969.741806] LustreError: 29656:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 5 previous similar messages [ 975.295115] LustreError: 29637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787135345 with bad export cookie 7668546084553887261 [ 975.309683] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 975.316688] LustreError: 29637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 975.927642] Lustre: server umount lustre-MDT0001 complete [ 991.284277] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 994.254991] Key type lgssc unregistered [ 994.491956] LNet: 33603:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 994.498326] LNetError: 33603:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 994.513748] LNet: Removed LNI 192.168.201.158@tcp [ 995.789277] Key type .llcrypt unregistered [ 995.791841] Key type ._llcrypt unregistered [ 1028.224393] Key type ._llcrypt registered [ 1028.226980] Key type .llcrypt registered [ 1028.394841] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 1030.108024] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1030.276845] alg: No test for adler32 (adler32-zlib) [ 1031.464745] Lustre: Lustre: Build Version: 2.17.57_11_g9d7c9d8 [ 1031.732173] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 1033.431209] Key type lgssc registered [ 1034.745767] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1043.012249] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1043.035768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1044.396245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1049.453848] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1054.698196] LustreError: 35028:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1054.724502] LustreError: 35028:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 2 previous similar messages [ 1058.801042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1059.138824] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1063.709584] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1071.415906] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1077.858802] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1085.074760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1085.583906] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1089.702388] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1092.059075] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1100.871572] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1105.025432] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1119.295524] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 06:31:27 (1787135487) [ 1121.514702] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1124.186452] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1131.938576] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 06:31:40 (1787135500) [ 1142.199754] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 06:31:50 (1787135510) [ 1143.561350] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1152.647787] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1152.651889] Lustre: Skipped 2 previous similar messages [ 1154.803966] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 06:32:03 (1787135523) [ 1156.708457] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1156.715243] Lustre: Skipped 2 previous similar messages [ 1165.263218] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1165.269585] Lustre: Skipped 4 previous similar messages [ 1180.605970] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 06:32:29 (1787135549) [ 1183.026845] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1186.667703] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1196.219822] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 06:32:44 (1787135564) [ 1198.519890] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1198.529566] Lustre: Skipped 6 previous similar messages [ 1213.875675] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 06:33:02 (1787135582) [ 1233.403071] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1233.406189] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1233.422356] LustreError: Skipped 1 previous similar message [ 1233.435268] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1233.443746] Lustre: Skipped 1 previous similar message [ 1233.465310] Lustre: Skipped 1 previous similar message [ 1238.505180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1238.512499] Lustre: Skipped 1 previous similar message [ 1243.632350] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1245.151116] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1245.295672] Lustre: server umount lustre-OST0000 complete [ 1248.742599] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1248.743155] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1248.758183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1248.758191] Lustre: Skipped 1 previous similar message [ 1253.857682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1253.867619] Lustre: Skipped 2 previous similar messages [ 1263.071155] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1263.321081] Lustre: server umount lustre-MDT0000 complete [ 1264.099221] LustreError: 35029:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1264.122755] LustreError: 35029:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 1 previous similar message [ 1267.665832] LustreError: 38591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787135637 with bad export cookie 2020063887712236608 [ 1267.676503] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1268.330868] Lustre: server umount lustre-MDT0001 complete [ 1283.269125] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 1286.315502] Key type lgssc unregistered [ 1286.648681] LNet: 42893:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1286.663694] LNetError: 42893:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1286.685572] LNet: Removed LNI 192.168.201.158@tcp [ 1287.620379] Key type .llcrypt unregistered [ 1287.623182] Key type ._llcrypt unregistered [ 1305.242970] Key type ._llcrypt registered [ 1305.246411] Key type .llcrypt registered [ 1305.401928] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 1323.430686] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 1324.840378] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1324.883389] alg: No test for adler32 (adler32-zlib) [ 1325.920980] Lustre: Lustre: Build Version: 2.17.57_11_g9d7c9d8 [ 1326.142853] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 1327.847282] Key type lgssc registered [ 1329.261449] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1369.514655] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1369.562882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1370.976319] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1371.025600] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1371.201272] Lustre: lustre-MDT0000: new disk, initializing [ 1371.440117] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1371.468531] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1377.665640] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1389.707788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1389.845739] Lustre: 46050:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1389.892501] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1389.895825] Lustre: Skipped 1 previous similar message [ 1389.955941] Lustre: lustre-MDT0001: new disk, initializing [ 1390.040574] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1390.090678] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1390.109237] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1395.320091] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1406.263854] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1412.174943] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1419.449709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1419.920600] Lustre: lustre-OST0000: new disk, initializing [ 1419.924300] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1419.933179] Lustre: 48019:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1420.034197] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1427.197592] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1428.567173] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1428.599494] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1428.725362] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1437.554902] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1442.513401] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1442.681334] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1447.255610] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1447.461183] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1454.563818] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1454.570506] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1454.571704] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1454.579372] Lustre: Skipped 1 previous similar message [ 1456.003517] Lustre: server umount lustre-OST0000 complete [ 1461.728729] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1461.745198] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1461.758135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1461.762619] Lustre: Skipped 1 previous similar message [ 1464.800276] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1464.804191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1464.835158] Lustre: Skipped 1 previous similar message [ 1466.315534] Lustre: server umount lustre-MDT0000 complete [ 1470.258447] LustreError: 46846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787135840 with bad export cookie 8389072464822158831 [ 1470.262148] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1470.267074] LustreError: 46846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1470.546053] Lustre: server umount lustre-MDT0001 complete [ 1480.259373] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 06:37:29 (1787135849) [ 1488.885559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1489.428078] LustreError: 50481:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1489.449513] LustreError: 50481:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 3 previous similar messages [ 1489.528457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1494.197217] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1495.014758] LustreError: 50482:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1499.106689] LustreError: 50481:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1502.134333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1502.423435] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1506.771811] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1514.307513] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1520.273939] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1528.164806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1528.578520] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1534.306598] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1542.387451] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1549.977311] LustreError: 53009:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1550.938938] Lustre: Modifying parameter general.timeout=1 in log params [ 1552.647250] Lustre: Modifying parameter general.timeout=3 in log params [ 1552.651920] Lustre: Skipped 1 previous similar message [ 1555.342138] Lustre: Modifying parameter general.timeout=6 in log params [ 1555.350335] Lustre: Skipped 2 previous similar messages [ 1559.518629] Lustre: Modifying parameter general.timeout=11 in log params [ 1559.525598] Lustre: Skipped 4 previous similar messages [ 1568.400951] Lustre: Modifying parameter general.timeout=21 in log params [ 1568.406598] Lustre: Skipped 9 previous similar messages [ 1585.029821] Lustre: Modifying parameter general.timeout=37 in log params [ 1585.036142] Lustre: Skipped 15 previous similar messages [ 1617.390167] Lustre: Modifying parameter general.timeout=74 in log params [ 1617.395990] Lustre: Skipped 36 previous similar messages [ 1677.264109] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 06:40:46 (1787136046) [ 1680.223234] Lustre: 59959:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1682.163308] Lustre: 60058:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1683.068792] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 in log params [ 1683.072796] Lustre: Skipped 59 previous similar messages [ 1685.584752] Lustre: 60271:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1685.595980] Lustre: 60271:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1716.626744] Lustre: 61835:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1716.631222] Lustre: 61835:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1716.654740] LustreError: 61835:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1721.082557] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 06:41:29 (1787136089) [ 1733.087915] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1733.090636] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1733.124463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1739.565031] Lustre: server umount lustre-MDT0000 complete [ 1741.280111] LustreError: 51209:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1741.301324] LustreError: 51209:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 2 previous similar messages [ 1744.038032] LustreError: 57816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787136114 with bad export cookie 8389072464822159713 [ 1744.044564] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1744.057243] LustreError: 57816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 1744.443511] Lustre: server umount lustre-MDT0001 complete [ 1759.636935] Lustre: server umount lustre-OST0000 complete [ 1788.873710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1789.156292] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1789.465632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1794.698452] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1803.663309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1803.842575] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1803.882426] Lustre: 64367:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1808.622553] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1816.830376] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1823.186298] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1832.120276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1832.493575] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1832.787536] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1832.798296] Lustre: Skipped 1 previous similar message [ 1839.974640] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1848.674358] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1850.771167] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1850.789402] Lustre: Skipped 19 previous similar messages [ 1850.826280] Lustre: 66935:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1859.552437] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1859.560561] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1859.569674] LustreError: Skipped 1 previous similar message [ 1859.576203] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1859.594375] Lustre: Skipped 3 previous similar messages [ 1859.598811] Lustre: Skipped 3 previous similar messages [ 1865.117752] Lustre: server umount lustre-OST0000 complete [ 1869.794920] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1869.813243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1869.823151] Lustre: Skipped 4 previous similar messages [ 1875.681329] Lustre: server umount lustre-MDT0000 complete [ 1880.040940] LustreError: 64378:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1880.064257] LustreError: 64378:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 1 previous similar message [ 1880.083741] LustreError: 64359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787136250 with bad export cookie 8389072464822162821 [ 1880.089987] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1880.099980] LustreError: 64359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1880.596291] Lustre: server umount lustre-MDT0001 complete [ 1893.205924] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 1896.344795] Key type lgssc unregistered [ 1896.599664] LNet: 68149:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1896.606291] LNetError: 68149:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1896.621298] LNet: Removed LNI 192.168.201.158@tcp [ 1897.480906] Key type .llcrypt unregistered [ 1897.491083] Key type ._llcrypt unregistered [ 1924.766646] Key type ._llcrypt registered [ 1924.769546] Key type .llcrypt registered [ 1924.892750] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 1926.339673] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1926.395493] alg: No test for adler32 (adler32-zlib) [ 1927.493552] Lustre: Lustre: Build Version: 2.17.57_11_g9d7c9d8 [ 1927.877245] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 1929.567359] Key type lgssc registered [ 1931.046683] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1939.181767] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1939.198432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1940.875316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1946.382703] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1950.185045] LustreError: 69564:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1950.208727] LustreError: 69564:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 2 previous similar messages [ 1955.058676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1955.297740] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1955.737181] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1960.998936] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1971.535839] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1979.518818] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1987.942544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1988.366798] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1995.525974] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2004.666809] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2023.536405] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2032.609214] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2032.613979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2032.635955] Lustre: Skipped 1 previous similar message [ 2032.650728] Lustre: Skipped 2 previous similar messages [ 2037.733129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2037.750125] Lustre: Skipped 2 previous similar messages [ 2038.342284] Lustre: server umount lustre-MDT0000 complete [ 2042.032749] LustreError: 70307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787136412 with bad export cookie 14782355203351903311 [ 2042.040876] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2042.046414] LustreError: 70307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2042.304855] Lustre: server umount lustre-MDT0001 complete [ 2056.816878] Lustre: server umount lustre-OST0000 complete [ 2087.126453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2087.510076] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2087.545886] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2087.892570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2093.812851] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2104.937048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2105.102214] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2105.140435] Lustre: 74458:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2105.814521] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2111.650138] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2121.240246] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2128.698764] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2136.833639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2136.981030] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2137.410611] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2144.971281] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2153.922950] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2155.915585] Lustre: 77036:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2158.048974] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2165.735777] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2165.743835] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2165.760044] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2165.764469] Lustre: Skipped 2 previous similar messages [ 2169.796276] Lustre: server umount lustre-OST0000 complete [ 2175.975347] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2175.979264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2175.990037] Lustre: Skipped 1 previous similar message [ 2176.014171] Lustre: Skipped 2 previous similar messages [ 2181.089057] LustreError: 74471:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2181.110370] LustreError: 74471:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 1 previous similar message [ 2182.888182] Lustre: server umount lustre-MDT0000 complete [ 2185.192158] LustreError: 74467:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2185.208016] LustreError: 74467:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 1 previous similar message [ 2186.866207] LustreError: 74452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787136556 with bad export cookie 14782355203351904151 [ 2186.875280] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2187.212451] Lustre: server umount lustre-MDT0001 complete [ 2200.294692] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 2203.680360] Key type lgssc unregistered [ 2203.929058] LNet: 78252:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2203.936156] LNetError: 78252:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2203.949655] LNet: Removed LNI 192.168.201.158@tcp [ 2204.956485] Key type .llcrypt unregistered [ 2204.961923] Key type ._llcrypt unregistered [ 2232.878700] Key type ._llcrypt registered [ 2232.881217] Key type .llcrypt registered [ 2232.985792] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 2234.695703] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2234.812528] alg: No test for adler32 (adler32-zlib) [ 2236.069439] Lustre: Lustre: Build Version: 2.17.57_11_g9d7c9d8 [ 2236.409260] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 2238.135340] Key type lgssc registered [ 2239.683431] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2248.699968] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2248.720886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2250.283621] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2255.092305] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2260.455986] LustreError: 79667:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2260.482612] LustreError: 79667:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 2 previous similar messages [ 2264.417535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2264.863696] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2269.779204] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2277.550277] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2285.076952] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2294.761487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2296.205540] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2305.252294] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2316.417698] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2323.001385] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2324.044777] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2325.721722] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2325.727189] Lustre: Skipped 1 previous similar message [ 2328.397362] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2328.408845] Lustre: Skipped 2 previous similar messages [ 2332.634396] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2332.638866] Lustre: Skipped 3 previous similar messages [ 2341.193553] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2341.215911] Lustre: Skipped 8 previous similar messages [ 2358.110986] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2358.113735] Lustre: Skipped 17 previous similar messages [ 2390.592368] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2390.596296] Lustre: Skipped 32 previous similar messages [ 2428.243409] Lustre: 87777:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2431.206634] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 06:53:20 (1787136800) [ 2433.018902] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2435.921327] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 06:53:24 (1787136804) [ 2442.724809] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2442.731943] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2442.746213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2442.767218] Lustre: Skipped 1 previous similar message [ 2446.653289] Lustre: server umount lustre-OST0000 complete [ 2452.960709] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2452.963364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2452.975544] Lustre: Skipped 2 previous similar messages [ 2452.995684] Lustre: Skipped 1 previous similar message [ 2456.778771] Lustre: server umount lustre-MDT0000 complete [ 2458.086605] LustreError: 79666:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2460.674830] LustreError: 84787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787136830 with bad export cookie 16097305335656354244 [ 2460.694980] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2460.695758] LustreError: 84787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2461.151471] Lustre: server umount lustre-MDT0001 complete [ 2473.872410] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 2477.716730] Key type lgssc unregistered [ 2478.043058] LNet: 89031:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2478.059591] LNetError: 89031:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2479.145792] LNet: Removed LNI 192.168.201.158@tcp [ 2480.100714] Key type .llcrypt unregistered [ 2480.103339] Key type ._llcrypt unregistered [ 2485.551185] Key type ._llcrypt registered [ 2485.557956] Key type .llcrypt registered [ 2485.674727] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2491.025525] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules [ 2492.603517] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2492.642654] alg: No test for adler32 (adler32-zlib) [ 2493.785561] Lustre: Lustre: Build Version: 2.17.57_11_g9d7c9d8 [ 2494.231499] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 2495.944702] Key type lgssc registered [ 2497.309133] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2498.339471] LustreError: 90241:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2517.917525] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 2519.733807] LustreError: 90241:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2526.590964] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2526.614958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2528.000377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2532.431924] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2537.443565] LustreError: 91255:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2537.476129] LustreError: 91255:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 2 previous similar messages [ 2542.576845] LustreError: 91254:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2543.069583] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 06:55:11 (1787136911) [ 2547.688667] LustreError: 91255:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2548.020436] LustreError: 91286:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2548.033232] LustreError: 91286:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2548.040348] LustreError: 91286:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 2548.396900] Lustre: server umount lustre-MDT0000 complete [ 2563.732094] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 2566.565605] Key type lgssc unregistered [ 2566.828892] LNet: 92783:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2566.842298] LNetError: 92783:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2567.925924] LNet: Removed LNI 192.168.201.158@tcp [ 2568.906538] Key type .llcrypt unregistered [ 2568.918765] Key type ._llcrypt unregistered [ 2593.452598] Key type ._llcrypt registered [ 2593.460715] Key type .llcrypt registered [ 2593.571178] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 2594.924260] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2594.934922] alg: No test for adler32 (adler32-zlib) [ 2596.285530] Lustre: Lustre: Build Version: 2.17.57_11_g9d7c9d8 [ 2596.626161] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 2598.415187] Key type lgssc registered [ 2600.514336] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2610.756608] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2610.768375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2612.215432] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2617.853896] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2622.433426] LustreError: 94003:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2622.445516] LustreError: 94003:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 2 previous similar messages [ 2627.422615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2627.557148] LustreError: 94004:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2628.051311] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2632.023803] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2640.685817] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2647.302654] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2655.035540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2655.856211] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2665.773964] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2677.346331] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2728.270973] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 06:58:16 (1787137096) [ 2730.314306] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2732.370527] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 06:58:21 (1787137101) [ 2739.172429] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2739.189708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2739.201205] Lustre: Skipped 1 previous similar message [ 2743.224957] Lustre: server umount lustre-MDT0000 complete [ 2744.298137] LustreError: 95976:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2744.328599] LustreError: 95976:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 2 previous similar messages [ 2747.268908] LustreError: 93984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787137117 with bad export cookie 9276924683382383734 [ 2747.272756] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2747.276501] LustreError: 93984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2747.825044] Lustre: server umount lustre-MDT0001 complete [ 2762.989834] Lustre: server umount lustre-OST0000 complete [ 2775.381516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2775.763242] LustreError: 98300:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2776.027052] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2780.705419] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2781.156581] LustreError: 98301:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2781.178996] LustreError: 98301:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 1 previous similar message [ 2790.371935] LustreError: 98300:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2790.393639] LustreError: 98300:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 1 previous similar message [ 2790.697441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2791.663527] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2795.979611] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2805.480466] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2811.901065] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2826.988600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2827.276271] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2827.285838] LustreError: 98290:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2827.347168] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2827.372250] LustreError: 100384:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2827.382187] LustreError: 100384:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2827.392713] LustreError: 100384:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2827.485372] Lustre: server umount lustre-OST0000 complete [ 2827.489665] LustreError: 100384:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2838.854626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2839.118501] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2839.129415] LustreError: 98290:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2839.171976] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2839.186813] LustreError: 100850:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2839.195311] LustreError: 100850:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2839.204844] LustreError: 100850:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2839.326383] Lustre: server umount lustre-OST0000 complete [ 2839.328509] LustreError: 100850:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2850.517348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2850.862352] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2850.865478] Lustre: Found index 0 for lustre-OST0000, updating log [ 2850.885345] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2851.082102] Lustre: lustre-OST0000: new disk, initializing [ 2851.087955] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2851.095588] Lustre: 101389:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2851.496153] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2857.647550] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2861.709509] Lustre: Failing over lustre-OST0000 [ 2861.858156] Lustre: server umount lustre-OST0000 complete [ 2868.193225] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2868.204843] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2868.207178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2868.207187] Lustre: Skipped 2 previous similar messages [ 2868.212160] Lustre: Skipped 2 previous similar messages [ 2871.777734] Lustre: server umount lustre-MDT0000 complete [ 2873.320571] LustreError: 98301:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2873.347459] LustreError: 98301:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 1 previous similar message [ 2876.210280] LustreError: 98281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787137246 with bad export cookie 9276924683382386485 [ 2876.220873] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2876.224345] LustreError: 98281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2876.593674] Lustre: server umount lustre-MDT0001 complete [ 2887.108977] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 07:00:55 (1787137255) [ 2906.186614] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 2916.271038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2916.669654] LustreError: 103748:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2916.849512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2921.412805] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2930.389196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2937.023663] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2940.836120] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2950.081754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2951.089352] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2951.095407] Lustre: Skipped 1 previous similar message [ 2958.655676] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2968.512648] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2968.606352] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2968.679838] Lustre: lustre-OST0001: new disk, initializing [ 2968.683979] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2968.688803] Lustre: 106065:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2976.128228] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2977.417683] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2977.440987] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2977.523974] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2984.553331] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2991.755184] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2998.242813] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2998.261991] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2998.274408] Lustre: Skipped 2 previous similar messages [ 2998.277146] Lustre: Skipped 2 previous similar messages [ 2998.300901] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 3002.111064] Lustre: server umount lustre-MDT0001 complete [ 3003.371104] LustreError: 105265:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3003.398553] LustreError: 105265:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 4 previous similar messages [ 3010.558772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3010.641616] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3010.646683] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3010.693929] Lustre: 103738:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3012.058594] LustreError: 107612:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3012.071291] LustreError: 107612:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3012.081867] LustreError: 103740:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3012.174841] LustreError: 107621:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.158@tcp: cfg command failed: rc = -17 [ 3012.183798] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3012.183798] [ 3016.234905] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3016.684225] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3025.458309] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 07:03:14 (1787137394) [ 3042.497858] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 3062.948457] Lustre: 109681:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3089.784732] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3094.105085] Lustre: 110817:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3185.638191] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3185.648317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3185.657426] Lustre: Skipped 2 previous similar messages [ 3185.684219] Lustre: Skipped 4 previous similar messages [ 3190.241627] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3190.248361] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3190.258825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3191.770137] Lustre: server umount lustre-MDT0000 complete [ 3195.878955] LustreError: 107577:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3195.895617] LustreError: 107577:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 8 previous similar messages [ 3196.734060] LustreError: 106064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787137566 with bad export cookie 9276924683382387479 [ 3196.755679] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3196.757411] LustreError: 106064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3197.329374] Lustre: server umount lustre-MDT0001 complete [ 3211.254418] Lustre: server umount lustre-OST0000 complete [ 3225.467875] Lustre: server umount lustre-OST0001 complete [ 3231.409807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3242.301556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3250.811055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3261.288660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3283.363759] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 3294.896865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3294.927705] Lustre: lustre-MDT0000: reset Object Index mappings [ 3295.426630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3295.433479] Lustre: Skipped 2 previous similar messages [ 3301.755313] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3310.273387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3310.293800] Lustre: lustre-MDT0001: reset Object Index mappings [ 3315.751193] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3318.878477] Lustre: 115921:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3326.650303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3335.017861] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3335.477207] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 3340.642275] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 3345.369044] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3351.585128] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3351.732129] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3353.231092] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3362.150575] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3365.961956] Lustre: 117799:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3422.591835] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 07:09:50 (1787137790) [ 3448.809052] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3448.817473] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3448.837616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3448.843948] Lustre: Skipped 3 previous similar messages [ 3454.127597] Lustre: server umount lustre-MDT0000 complete [ 3456.996962] LustreError: 114773:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3457.036869] LustreError: 114773:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 9 previous similar messages [ 3458.514137] LustreError: 115923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787137828 with bad export cookie 9276924683382434379 [ 3458.518178] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3458.525570] LustreError: 115923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3465.252985] Lustre: server umount lustre-MDT0001 complete [ 3484.127175] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3484.397811] Lustre: server umount lustre-OST0000 complete [ 3495.795182] Lustre: server umount lustre-OST0001 complete [ 3504.255646] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 3516.085789] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 3554.787119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3555.062199] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3555.101833] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3555.164627] Lustre: lustre-MDT0000: new disk, initializing [ 3555.360351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3555.371482] Lustre: Skipped 3 previous similar messages [ 3555.424496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3560.266984] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3568.047652] Lustre: server umount lustre-MDT0000 complete [ 3579.727666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3580.014825] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3580.027524] Lustre: 122069:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3580.035968] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3584.963511] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3589.807856] Lustre: server umount lustre-MDT0000 complete [ 3600.162140] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 07:12:48 (1787137968) [ 3602.710818] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3605.709530] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 07:12:53 (1787137973) [ 3629.123378] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 3641.198188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3648.849642] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3658.670061] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3658.812567] Lustre: 124120:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3658.892535] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3658.898579] Lustre: Skipped 1 previous similar message [ 3659.012274] Lustre: lustre-MDT0001: new disk, initializing [ 3659.148412] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3659.170706] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3664.176477] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3669.588951] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3676.958980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3677.182195] Lustre: lustre-OST0000: new disk, initializing [ 3677.185284] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3677.191294] Lustre: 125702:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3678.746207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3678.756225] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3678.906965] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3684.980989] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3697.382428] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3697.564423] Lustre: lustre-OST0001: new disk, initializing [ 3697.574492] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3697.580554] Lustre: 126574:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3699.297347] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3699.334485] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3699.438706] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3705.250267] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3715.565360] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3719.065564] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4122.484726] Lustre: 133001:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4122.499993] Lustre: 133001:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4122.502725] Lustre: 133001:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4122.510991] Lustre: 133001:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4122.526909] Lustre: 133001:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4122.534803] Lustre: 133001:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4398.855280] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 07:26:06 (1787138766) [ 4416.493157] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4416.498060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4416.510077] Lustre: Skipped 6 previous similar messages [ 4416.520216] Lustre: Skipped 8 previous similar messages [ 4421.615161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4421.635148] Lustre: Skipped 3 previous similar messages [ 4422.094952] Lustre: server umount lustre-MDT0000 complete [ 4426.511168] LustreError: 124862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787138796 with bad export cookie 9276924683382503224 [ 4426.523500] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4426.550823] LustreError: 124862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4426.733693] LustreError: 124127:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4426.754485] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4426.755769] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4426.775650] LustreError: 124127:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 5 previous similar messages [ 4426.804665] Lustre: Skipped 3 previous similar messages [ 4433.613331] Lustre: server umount lustre-MDT0001 complete [ 4444.297796] Lustre: server umount lustre-OST0000 complete [ 4455.809732] Lustre: server umount lustre-OST0001 complete [ 4463.294793] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 4474.224610] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 4514.464453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4514.809777] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4514.841620] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4514.936507] Lustre: lustre-MDT0000: new disk, initializing [ 4515.038973] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4515.044493] Lustre: Skipped 5 previous similar messages [ 4515.068380] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4519.558181] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4530.500146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4530.633654] Lustre: 136053:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4530.699449] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4530.703793] Lustre: Skipped 1 previous similar message [ 4530.784341] Lustre: lustre-MDT0001: new disk, initializing [ 4530.950914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4530.972195] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4535.718041] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4546.277918] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4552.930414] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4561.642686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4562.070149] Lustre: lustre-OST0000: new disk, initializing [ 4562.079546] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4562.096933] Lustre: 138022:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4562.300334] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4562.311702] Lustre: Skipped 1 previous similar message [ 4563.569544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4563.589754] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4563.753112] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4570.735985] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4584.375237] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4596.155760] Lustre: lustre-MDD0000: changelog on [ 4599.898567] Lustre: lustre-MDD0001: changelog on [ 6737.810309] Lustre: lustre-MDD0001: changelog off [ 6739.435276] Lustre: 136061:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6740.230344] Lustre: lustre-MDD0000: changelog off [ 6748.138646] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6748.156614] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6748.174082] Lustre: Skipped 1 previous similar message [ 6748.192571] Lustre: Skipped 4 previous similar messages [ 6751.856343] Lustre: server umount lustre-OST0000 complete [ 6758.370351] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6758.374547] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6758.382441] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6758.418872] Lustre: Skipped 1 previous similar message [ 6762.004606] Lustre: server umount lustre-MDT0000 complete [ 6763.492270] LustreError: 136918:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6763.513673] LustreError: 136918:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 3 previous similar messages [ 6765.695245] LustreError: 136045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787141135 with bad export cookie 9276924683383283241 [ 6765.710769] LustreError: 136045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6765.719839] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6766.021299] Lustre: server umount lustre-MDT0001 complete [ 6777.881123] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 6780.647755] Key type lgssc unregistered [ 6781.027622] LNet: 147203:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6781.044848] LNetError: 147203:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6781.074359] LNet: Removed LNI 192.168.201.158@tcp [ 6781.905198] Key type .llcrypt unregistered [ 6781.906943] Key type ._llcrypt unregistered [ 6802.793933] Key type ._llcrypt registered [ 6802.801672] Key type .llcrypt registered [ 6802.964907] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 6820.755965] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 6822.283931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6822.344414] alg: No test for adler32 (adler32-zlib) [ 6823.472072] Lustre: Lustre: Build Version: 2.17.57_11_g9d7c9d8 [ 6823.716792] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 6825.431151] Key type lgssc registered [ 6826.916622] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6865.068737] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6865.180090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6866.551929] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6866.586609] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6866.743936] Lustre: lustre-MDT0000: new disk, initializing [ 6866.863616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6866.887757] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6872.148611] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 6886.464867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6886.591816] Lustre: 150410:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6886.652167] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6886.657342] Lustre: Skipped 1 previous similar message [ 6886.728592] Lustre: lustre-MDT0001: new disk, initializing [ 6886.819347] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6886.861258] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6886.881295] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6893.318608] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 6904.628982] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6910.585650] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6918.135726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6918.426988] Lustre: lustre-OST0000: new disk, initializing [ 6918.432259] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6918.440405] Lustre: 152379:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6918.532420] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6924.844686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6924.860250] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6925.020990] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6925.266378] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 6934.913616] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6940.884179] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_module obdecho/obdecho [ 6942.809482] LustreError: 153382:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6942.817704] LustreError: 153382:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6944.681887] LustreError: 153478:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6958.560148] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6958.565319] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6958.578636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6961.124027] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6961.124115] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6961.129396] Lustre: Skipped 1 previous similar message [ 6961.141161] Lustre: Skipped 1 previous similar message [ 6962.889071] Lustre: server umount lustre-MDT0000 complete [ 6966.945377] LustreError: 150402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787141336 with bad export cookie 4722181555131088561 [ 6966.955782] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6967.306857] Lustre: server umount lustre-MDT0001 complete [ 6982.800542] Lustre: server umount lustre-OST0000 complete [ 7011.584795] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 7027.879843] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 7072.921451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7073.273995] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7073.308323] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7073.418404] Lustre: lustre-MDT0000: new disk, initializing [ 7073.526798] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7073.560443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7078.128438] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 7092.428722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7092.665761] Lustre: 157218:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7092.734579] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7092.741277] Lustre: Skipped 1 previous similar message [ 7092.822438] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7092.824976] Lustre: Skipped 1 previous similar message [ 7092.897566] Lustre: lustre-MDT0001: new disk, initializing [ 7093.047629] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7093.113678] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7093.131183] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7099.306071] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 7114.556929] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7122.534241] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7132.386152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7132.740758] Lustre: lustre-OST0000: new disk, initializing [ 7132.749951] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7132.761332] Lustre: 159189:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7132.912586] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7139.895607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7139.915933] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7140.032474] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7141.553543] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 7153.109315] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7159.057395] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7159.281880] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7164.658423] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7164.973312] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7171.041862] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7171.048831] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7171.063339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7171.064523] Lustre: Skipped 1 previous similar message [ 7171.077102] Lustre: Skipped 1 previous similar message [ 7176.167382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7176.175397] Lustre: Skipped 1 previous similar message [ 7181.279164] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7181.480517] Lustre: server umount lustre-OST0000 complete [ 7186.400645] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7186.421933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7186.432646] Lustre: Skipped 3 previous similar messages [ 7192.072818] Lustre: server umount lustre-MDT0000 complete [ 7196.405776] LustreError: 157210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787141566 with bad export cookie 4722181555131089513 [ 7196.410598] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7196.418715] LustreError: 157210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7196.877631] Lustre: server umount lustre-MDT0001 complete [ 7200.911962] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 08:12:49 (1787141569) [ 7203.178637] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7205.519685] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 08:12:53 (1787141573) [ 7214.216641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7214.648470] LustreError: 161601:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7214.673168] LustreError: 161601:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 4 previous similar messages [ 7214.737781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7219.289675] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 7220.200324] LustreError: 161602:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7224.300035] LustreError: 161601:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7228.155936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7232.864296] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 7241.139686] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7248.613295] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7255.997227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7256.417389] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7256.426921] Lustre: Skipped 1 previous similar message [ 7263.907863] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 7273.025173] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7284.193871] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7284.199324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7284.210731] Lustre: Skipped 2 previous similar messages [ 7284.226019] Lustre: Skipped 4 previous similar messages [ 7286.291708] Lustre: server umount lustre-MDT0000 complete [ 7289.318767] LustreError: 161601:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7289.354413] LustreError: 161601:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 2 previous similar messages [ 7290.690557] LustreError: 161583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787141660 with bad export cookie 4722181555131090402 [ 7290.698596] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7290.719618] LustreError: 161583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7294.432895] LustreError: 161596:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7294.433639] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7294.456547] LustreError: 161596:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 1 previous similar message [ 7294.467921] Lustre: Skipped 1 previous similar message [ 7305.034965] LustreError: 161598:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7305.056548] LustreError: 161598:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 4 previous similar messages [ 7305.183182] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7305.482542] Lustre: server umount lustre-MDT0001 complete [ 7312.840380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7327.162471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7327.456215] LustreError: 165448:0:(llog.c:1655:llog_backup()) MGC192.168.201.158@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7327.462848] Lustre: 165448:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.158@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7330.272583] LustreError: 165453:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7330.293615] LustreError: 165453:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 1 previous similar message [ 7366.115411] LustreError: 165455:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7366.142341] LustreError: 165455:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 27 previous similar messages [ 7388.319649] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7388.338274] Lustre: 165448:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.158@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7429.132857] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7430.993374] LustreError: 165455:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7431.014032] LustreError: 165455:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 43 previous similar messages [ 7434.218533] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7434.799310] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 7446.108339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7455.718271] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7455.727424] LustreError: Skipped 1 previous similar message [ 7460.840619] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x41888f1aee7d9f3b [ 7460.862521] Lustre: MGC192.168.201.158@tcp: Connection restored to 0@lo (at 0@lo) [ 7466.197868] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 7466.506198] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7475.266814] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7477.099618] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7484.370439] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7486.165506] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7507.433577] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7507.454514] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7507.466874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7507.466884] Lustre: Skipped 1 previous similar message [ 7507.478910] Lustre: Skipped 7 previous similar messages [ 7512.609663] Lustre: server umount lustre-MDT0000 complete [ 7517.056920] LustreError: 161583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787141887 with bad export cookie 4722181555131096891 [ 7517.098125] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7517.101232] LustreError: 161583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7517.731765] Lustre: server umount lustre-MDT0001 complete [ 7532.502392] Lustre: server umount lustre-OST0000 complete [ 7563.400603] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 7579.226032] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 7622.783816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7623.034606] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7623.065795] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7623.160747] Lustre: lustre-MDT0000: new disk, initializing [ 7623.328541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7623.336629] Lustre: Skipped 1 previous similar message [ 7623.357309] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7628.476857] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 7641.063412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7641.257206] Lustre: 170581:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7641.575597] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7647.622101] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 7657.788348] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7665.484703] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7674.419317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7674.643717] Lustre: lustre-OST0000: new disk, initializing [ 7674.646689] Lustre: Skipped 1 previous similar message [ 7674.649457] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7674.656575] Lustre: Skipped 2 previous similar messages [ 7674.662469] Lustre: 172552:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7681.119409] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7681.138384] Lustre: Skipped 1 previous similar message [ 7681.156392] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7681.334353] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7681.926060] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 7695.406908] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7701.488082] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7701.694819] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7706.742648] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7706.935367] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7712.235681] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7712.237314] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7712.245211] LustreError: Skipped 1 previous similar message [ 7712.248977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7712.248985] Lustre: Skipped 2 previous similar messages [ 7712.288704] Lustre: Skipped 2 previous similar messages [ 7715.392446] Lustre: server umount lustre-OST0000 complete [ 7726.044324] Lustre: server umount lustre-MDT0000 complete [ 7727.584323] LustreError: 172559:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7727.602756] LustreError: 172559:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 30 previous similar messages [ 7730.396364] LustreError: 170574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787142100 with bad export cookie 4722181555131113684 [ 7730.400551] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7730.405319] LustreError: 170574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7730.958393] Lustre: server umount lustre-MDT0001 complete [ 7741.155433] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 08:21:49 (1787142109) [ 7748.762501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7754.816588] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 7764.673136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7770.174364] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 7781.377542] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7791.565144] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7802.306538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7812.397380] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 7821.572133] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7834.547948] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 08:23:23 (1787142203) [ 7840.224822] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7840.237522] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7840.243391] LustreError: Skipped 1 previous similar message [ 7840.266103] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7840.276338] Lustre: Skipped 3 previous similar messages [ 7840.296645] Lustre: Skipped 4 previous similar messages [ 7845.740405] Lustre: server umount lustre-OST0000 complete [ 7856.712565] Lustre: server umount lustre-MDT0000 complete [ 7861.446208] LustreError: 174946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787142231 with bad export cookie 4722181555131114573 [ 7861.467713] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7862.182455] Lustre: server umount lustre-MDT0001 complete [ 7875.919224] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 7879.204375] Key type lgssc unregistered [ 7879.600562] LNet: 178703:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7879.627277] LNetError: 178703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7879.644607] LNet: Removed LNI 192.168.201.158@tcp [ 7880.881453] Key type .llcrypt unregistered [ 7880.883824] Key type ._llcrypt unregistered [ 7903.627682] Key type ._llcrypt registered [ 7903.632648] Key type .llcrypt registered [ 7903.777771] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 7905.328316] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7905.359582] alg: No test for adler32 (adler32-zlib) [ 7906.576580] Lustre: Lustre: Build Version: 2.17.57_11_g9d7c9d8 [ 7906.822854] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 7908.511288] Key type lgssc registered [ 7909.932404] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7918.741567] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7918.784534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7934.433550] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7939.553425] LustreError: 179946:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.158@tcp: failed processing log, type 4: rc = -110 [ 7944.671557] LustreError: 179946:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7944.686290] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7944.697872] LustreError: MGC192.168.201.158@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7944.726437] LustreError: 179946:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7944.734756] LustreError: 179946:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7944.747232] LustreError: 179946:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7944.850104] Lustre: server umount lustre-OST0000 complete [ 7944.864549] LustreError: 179946:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7954.265822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7955.775923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7960.715574] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 7965.157203] LustreError: 180446:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7965.176560] LustreError: 180446:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 2 previous similar messages [ 7969.910953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7970.425624] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7975.678450] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 7984.240588] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7991.136966] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7998.421345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7998.621464] LustreError: 182261:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7998.631876] Lustre: 182261:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.201.158@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7998.814897] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8005.589270] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 8013.897803] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8021.476901] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8021.483756] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8021.484783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8021.498986] Lustre: Skipped 1 previous similar message [ 8022.907110] Lustre: server umount lustre-MDT0000 complete [ 8026.603845] LustreError: 180445:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8026.623602] LustreError: 180445:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 2 previous similar messages [ 8027.003050] LustreError: 180425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787142396 with bad export cookie 3414239079624775182 [ 8027.013945] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8027.021343] LustreError: 180425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8027.356622] Lustre: server umount lustre-MDT0001 complete [ 8041.802440] Lustre: server umount lustre-OST0000 complete [ 8051.853891] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 8062.816966] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 8103.800346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8104.139281] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8104.238990] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8104.381847] Lustre: lustre-MDT0000: new disk, initializing [ 8104.549133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8104.580077] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8110.582433] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 8121.436146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8121.596442] Lustre: 185797:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8121.641923] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8121.648495] Lustre: Skipped 1 previous similar message [ 8121.711316] Lustre: lustre-MDT0001: new disk, initializing [ 8121.817778] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8121.875238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8121.885075] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8126.058764] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 8138.031891] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8147.285590] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8156.461854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8156.827627] Lustre: lustre-OST0000: new disk, initializing [ 8156.832311] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8156.844367] Lustre: 187765:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8156.952741] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8163.735212] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 8163.903888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8163.914894] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8163.998856] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8174.747051] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8180.897229] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8181.366518] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8188.449934] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8188.780829] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8194.530341] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8194.536577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8194.555405] Lustre: Skipped 2 previous similar messages [ 8194.581196] Lustre: Skipped 3 previous similar messages [ 8199.652362] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8199.663381] Lustre: Skipped 1 previous similar message [ 8204.793626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8204.800489] Lustre: Skipped 1 previous similar message [ 8205.279244] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8205.622163] Lustre: server umount lustre-OST0000 complete [ 8209.891101] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8209.905216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8209.912917] Lustre: Skipped 1 previous similar message [ 8220.133606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8220.137701] Lustre: Skipped 3 previous similar messages [ 8224.224695] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8224.535280] Lustre: server umount lustre-MDT0000 complete [ 8225.255161] LustreError: 185808:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8225.274663] LustreError: 185808:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 1 previous similar message [ 8228.804712] LustreError: 185788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787142598 with bad export cookie 3414239079624775903 [ 8228.821075] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8229.128379] Lustre: server umount lustre-MDT0001 complete [ 8238.238725] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 08:30:06 (1787142606) [ 8257.082490] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 8260.536348] Key type lgssc unregistered [ 8260.956242] LNet: 190715:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8260.963235] LNetError: 190715:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8260.978442] LNet: Removed LNI 192.168.201.158@tcp [ 8262.182159] Key type .llcrypt unregistered [ 8262.184189] Key type ._llcrypt unregistered [ 8285.456725] Key type ._llcrypt registered [ 8285.462573] Key type .llcrypt registered [ 8285.592829] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 8286.810280] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8286.923609] alg: No test for adler32 (adler32-zlib) [ 8288.215922] Lustre: Lustre: Build Version: 2.17.57_11_g9d7c9d8 [ 8288.545552] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 8290.239158] Key type lgssc registered [ 8291.081366] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8298.573794] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8298.595147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8300.102728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8305.745742] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 8310.242691] LustreError: 191941:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8310.286190] LustreError: 191941:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 2 previous similar messages [ 8315.376750] LustreError: 191942:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8316.067538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8316.433330] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8322.341868] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 8332.170945] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8339.714411] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8349.309876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8349.675463] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8349.680645] LustreError: 193806:0:(llog.c:1696:llog_backup()) MGC192.168.201.158@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8349.697802] Lustre: 193806:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.201.158@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8349.909277] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8357.245965] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 8367.406478] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8372.704372] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8372.711917] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8372.731613] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8375.778991] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8375.783222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8375.817970] Lustre: Skipped 1 previous similar message [ 8375.838375] Lustre: Skipped 2 previous similar messages [ 8377.741359] Lustre: server umount lustre-MDT0000 complete [ 8380.908186] LustreError: 191941:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8380.954616] LustreError: 191941:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 2 previous similar messages [ 8381.823401] LustreError: 191921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787142751 with bad export cookie 5709457722283459396 [ 8381.840764] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8382.489529] Lustre: server umount lustre-MDT0001 complete [ 8397.602664] Lustre: server umount lustre-OST0000 complete [ 8408.914717] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 8421.654710] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 8458.379803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8458.645509] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8458.669082] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8458.728514] Lustre: lustre-MDT0000: new disk, initializing [ 8458.792372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8458.815338] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8463.499185] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 8474.466315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8474.558810] Lustre: 197334:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8474.597542] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8474.599510] Lustre: Skipped 1 previous similar message [ 8474.601785] LustreError: 198117:0:(llog.c:1696:llog_backup()) MGC192.168.201.158@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8474.607076] LustreError: 198117:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8474.611098] Lustre: 198117:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.158@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8474.626943] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8474.630462] Lustre: Skipped 1 previous similar message [ 8474.681747] Lustre: lustre-MDT0001: new disk, initializing [ 8474.749637] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8474.814597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8474.822573] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8479.447633] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 8491.061544] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8498.125205] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8505.569250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8505.843380] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8505.846065] Lustre: Skipped 1 previous similar message [ 8505.849788] LustreError: 199278:0:(llog.c:1696:llog_backup()) MGC192.168.201.158@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8505.864196] LustreError: 199278:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8505.870504] Lustre: 199278:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.158@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8505.879746] Lustre: 199278:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8505.906492] Lustre: lustre-OST0000: new disk, initializing [ 8505.915270] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8505.927512] Lustre: 199301:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8506.016236] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8513.150049] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 8513.162780] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8513.186131] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8513.296083] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8524.044709] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8529.066430] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8529.276792] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8534.241352] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8534.478862] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8537.056470] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8537.060439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8537.074655] Lustre: Skipped 1 previous similar message [ 8537.088239] Lustre: Skipped 1 previous similar message [ 8542.184048] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8542.433238] Lustre: server umount lustre-OST0000 complete [ 8546.784732] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8546.793397] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8546.804382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8546.810251] Lustre: Skipped 1 previous similar message [ 8552.309611] Lustre: server umount lustre-MDT0000 complete [ 8552.430531] LustreError: 197342:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8552.450938] LustreError: 197342:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 1 previous similar message [ 8556.958627] LustreError: 197325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787142926 with bad export cookie 5709457722283460117 [ 8556.968192] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8556.972826] LustreError: 197325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 8557.301305] Lustre: server umount lustre-MDT0001 complete [ 8567.750546] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 08:35:36 (1787142936) [ 8589.332252] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 8603.082477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8603.433311] LustreError: 202448:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8603.486630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8608.903834] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 8613.865726] LustreError: 202449:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8613.877513] LustreError: 202449:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 1 previous similar message [ 8618.548884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8623.888863] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 8627.250639] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8636.455827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8637.128401] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8637.136633] Lustre: Skipped 1 previous similar message [ 8645.453489] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 8656.188090] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8656.521369] Lustre: lustre-OST0001: new disk, initializing [ 8656.532403] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8656.542364] Lustre: 204757:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8662.623490] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8662.634496] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8662.739700] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8664.389457] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 8676.819424] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8683.116919] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8686.035159] Lustre: DEBUG MARKER: ADD OST3 [ 8691.942848] loop: module loaded [ 8692.094272] loop0: detected capacity change from 0 to 204800000 [ 8692.378721] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8692.392927] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8692.419283] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8699.818337] loop0: detected capacity change from 0 to 204800000 [ 8699.879380] Lustre: DEBUG MARKER: STOP OST3 [ 8706.961508] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8707.346916] Lustre: lustre-OST0003: new disk, initializing [ 8707.349959] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8707.362127] Lustre: 206795:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8707.536752] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8707.539870] Lustre: Skipped 1 previous similar message [ 8709.168616] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8709.449244] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8709.461471] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8709.532908] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8711.200489] LustreError: 206937:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8711.327528] Lustre: Failing over lustre-OST0003 [ 8711.712302] Lustre: server umount lustre-OST0003 complete [ 8714.620778] LustreError: 203933:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.201.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8714.721514] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8714.741321] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8714.758033] Lustre: Skipped 1 previous similar message [ 8716.338954] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 8724.408480] Lustre: DEBUG MARKER: START OST3 again [ 8731.961271] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8732.251629] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8733.943546] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8734.291605] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8734.305465] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8738.391325] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 8761.016078] Lustre: Failing over lustre-OST0003 [ 8761.085899] Lustre: server umount lustre-OST0003 complete [ 8762.862742] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8762.867927] LustreError: 203934:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8762.873747] Lustre: Skipped 1 previous similar message [ 8762.895230] LustreError: 203934:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 12 previous similar messages [ 8782.820794] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8782.828230] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8782.838747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8782.841909] Lustre: Skipped 2 previous similar messages [ 8785.863622] Lustre: server umount lustre-MDT0000 complete [ 8790.063348] LustreError: 202429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787143160 with bad export cookie 5709457722283460999 [ 8790.073719] LustreError: 202429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8790.079578] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8796.578272] Lustre: server umount lustre-MDT0001 complete [ 8801.663510] Lustre: server umount lustre-OST0000 complete [ 8806.415395] Lustre: server umount lustre-OST0001 complete [ 8814.907849] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 8826.259073] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 8866.335190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8866.627850] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8866.665494] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8866.757996] Lustre: lustre-MDT0000: new disk, initializing [ 8866.882755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8866.893642] Lustre: Skipped 1 previous similar message [ 8866.925889] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8871.536682] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 8884.103778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8884.301983] Lustre: 212219:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8884.422775] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8884.443534] Lustre: Skipped 1 previous similar message [ 8884.595541] Lustre: lustre-MDT0001: new disk, initializing [ 8884.793927] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8884.800459] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8889.262822] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 8900.016617] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8906.046540] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8914.975357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8915.203923] Lustre: 214191:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8917.050345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8917.061524] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8917.244713] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8923.728563] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 8936.416979] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8943.705994] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8943.889944] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8949.730934] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8950.009030] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8953.312302] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8953.321367] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8953.326259] LustreError: Skipped 1 previous similar message [ 8953.328337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8953.328343] Lustre: Skipped 8 previous similar messages [ 8953.375542] Lustre: Skipped 6 previous similar messages [ 8958.832691] Lustre: server umount lustre-OST0000 complete [ 8977.375933] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8977.686913] Lustre: server umount lustre-MDT0000 complete [ 8978.914876] LustreError: 212226:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8978.933820] LustreError: 212226:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 17 previous similar messages [ 8982.083084] LustreError: 213023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787143352 with bad export cookie 5709457722283463295 [ 8982.105179] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8982.542613] Lustre: server umount lustre-MDT0001 complete [ 8986.687708] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 08:42:35 (1787143355) [ 9011.192871] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 9027.308854] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 9069.935649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9070.218538] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9070.257475] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9070.263213] Lustre: Skipped 1 previous similar message [ 9070.375485] Lustre: lustre-MDT0000: new disk, initializing [ 9070.387933] Lustre: Skipped 1 previous similar message [ 9070.536250] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9075.347766] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 9088.504033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9088.648487] Lustre: 218824:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9088.891855] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9094.336798] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 9104.877445] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9112.228507] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9119.972967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9120.201039] Lustre: 220793:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9121.964893] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9122.116071] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9126.999971] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 9138.331506] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9144.044895] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9144.390810] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9150.302211] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9150.513152] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9152.992410] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9153.009977] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9153.019344] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9153.030538] Lustre: Skipped 3 previous similar messages [ 9153.048842] Lustre: Skipped 10 previous similar messages [ 9159.045793] Lustre: server umount lustre-OST0000 complete [ 9169.764865] Lustre: server umount lustre-MDT0000 complete [ 9173.398038] LustreError: 218814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787143543 with bad export cookie 5709457722283464177 [ 9173.403965] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9173.424817] LustreError: 218814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 9173.929555] Lustre: server umount lustre-MDT0001 complete [ 9184.923402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9185.538378] LustreError: 223010:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9185.653517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9185.676788] Lustre: Skipped 5 previous similar messages [ 9191.259500] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 9203.230938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9209.807042] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 9218.437208] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9225.169399] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9232.830934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9240.756985] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 9251.111127] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9269.216683] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9269.240564] LustreError: Skipped 1 previous similar message [ 9269.250931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9269.263314] Lustre: Skipped 6 previous similar messages [ 9274.103743] Lustre: server umount lustre-OST0000 complete [ 9281.519386] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9281.551483] Lustre: Skipped 5 previous similar messages [ 9285.034964] Lustre: server umount lustre-MDT0000 complete [ 9289.621293] LustreError: 222991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787143659 with bad export cookie 5709457722283465059 [ 9289.628104] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9289.632312] LustreError: 222991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9290.082371] Lustre: server umount lustre-MDT0001 complete [ 9393.090650] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 9396.267609] Key type lgssc unregistered [ 9396.707053] LNet: 226702:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9396.721511] LNetError: 226702:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9396.740226] LNet: Removed LNI 192.168.201.158@tcp [ 9397.665248] Key type .llcrypt unregistered [ 9397.669575] Key type ._llcrypt unregistered [ 9426.292611] Key type ._llcrypt registered [ 9426.295508] Key type .llcrypt registered [ 9426.499555] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 9443.791346] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 9445.144963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9445.176173] alg: No test for adler32 (adler32-zlib) [ 9446.268353] Lustre: Lustre: Build Version: 2.17.57_11_g9d7c9d8 [ 9446.533959] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 9448.271192] Key type lgssc registered [ 9449.610466] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9491.766175] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9491.799656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9493.126765] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9493.163590] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9493.266339] Lustre: lustre-MDT0000: new disk, initializing [ 9493.395067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9493.414064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9498.805771] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 9511.655247] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9511.860885] Lustre: 230056:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9511.993547] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9512.005891] Lustre: Skipped 1 previous similar message [ 9512.115490] Lustre: lustre-MDT0001: new disk, initializing [ 9512.232928] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9512.285067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9512.301201] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9517.079854] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 9527.556484] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9534.697390] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9542.906524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9543.454585] Lustre: lustre-OST0000: new disk, initializing [ 9543.462604] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9543.467662] Lustre: 232023:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9543.583989] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9549.969279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9549.990394] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9550.193050] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9551.848595] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 9563.487258] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9569.637490] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9569.945633] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9574.709324] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9575.044881] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9581.034996] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9581.038073] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9581.051141] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9581.070214] Lustre: Skipped 1 previous similar message [ 9581.090238] Lustre: Skipped 1 previous similar message [ 9583.638447] Lustre: server umount lustre-OST0000 complete [ 9589.217260] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9589.230391] LustreError: Skipped 1 previous similar message [ 9589.232725] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9589.252097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9591.267458] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9591.269091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9591.313937] Lustre: Skipped 1 previous similar message [ 9594.675709] Lustre: server umount lustre-MDT0000 complete [ 9598.307846] LustreError: 230049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787143968 with bad export cookie 255252056052016000 [ 9598.311388] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9598.314239] LustreError: 230049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9598.759914] Lustre: server umount lustre-MDT0001 complete [ 9620.878648] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 9635.083452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9636.003624] LustreError: 234961:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9636.034772] LustreError: 234961:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 3 previous similar messages [ 9636.155474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9641.440642] LustreError: 234962:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9642.173684] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 9646.572347] LustreError: 234961:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9651.707915] LustreError: 234962:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9656.423365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9656.801142] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9657.175902] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9664.802365] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 9667.930716] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9677.702883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9678.379916] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9686.161568] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 9696.508240] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9696.669560] Lustre: lustre-OST0001: new disk, initializing [ 9696.674430] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9696.682969] Lustre: 237271:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9696.789762] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9702.076480] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9702.084221] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9702.198940] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9704.235890] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 9715.645391] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9724.300138] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9738.981221] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 08:55:07 (1787144107) [ 9743.332823] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9743.344205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9743.365473] Lustre: Skipped 2 previous similar messages [ 9743.376926] Lustre: Skipped 1 previous similar message [ 9748.068487] Lustre: server umount lustre-MDT0000 complete [ 9748.452445] LustreError: 234962:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9748.468968] LustreError: 234962:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 3 previous similar messages [ 9751.974668] LustreError: 234942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787144121 with bad export cookie 255252056052016882 [ 9751.991298] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9751.994321] LustreError: 234942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9752.305592] Lustre: server umount lustre-MDT0001 complete [ 9766.388394] Lustre: server umount lustre-OST0000 complete [ 9769.249376] Lustre: 228281:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787144123/real 1787144123] req@ffff8d0832294e00 x1873956114125312/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787144139 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9769.274108] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9769.285769] Lustre: Skipped 1 previous similar message [ 9770.983781] Lustre: server umount lustre-OST0001 complete [ 9778.522347] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 9792.069904] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 9836.551307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9836.885613] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9836.950297] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9837.102243] Lustre: lustre-MDT0000: new disk, initializing [ 9837.268557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9837.315977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9842.506509] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 9854.060632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9854.174318] Lustre: 241216:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9854.226690] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9854.230177] Lustre: Skipped 1 previous similar message [ 9854.287209] Lustre: lustre-MDT0001: new disk, initializing [ 9854.391511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9854.404568] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9859.149775] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 9870.782791] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9877.109592] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9884.719507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9885.084530] Lustre: lustre-OST0000: new disk, initializing [ 9885.093881] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9885.120619] Lustre: 243185:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9886.228970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9886.249834] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9886.370778] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9893.061816] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 9904.474879] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9910.791384] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9911.007217] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9917.173367] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9917.549686] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9922.019368] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9922.040134] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9922.072347] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9922.092803] Lustre: Skipped 2 previous similar messages [ 9925.953721] Lustre: server umount lustre-OST0000 complete [ 9931.236948] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9931.241842] LustreError: Skipped 1 previous similar message [ 9936.147116] Lustre: server umount lustre-MDT0000 complete [ 9937.900301] LustreError: 242095:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9937.915303] LustreError: 242095:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 1 previous similar message [ 9940.541287] LustreError: 242012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787144310 with bad export cookie 255252056052019017 [ 9940.545334] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9940.572116] LustreError: 242012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9940.992569] Lustre: server umount lustre-MDT0001 complete [ 9950.733891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9951.224485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9951.227907] Lustre: Skipped 2 previous similar messages [ 9956.182834] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 9956.323833] LustreError: 245402:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9956.349292] LustreError: 245402:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 1 previous similar message [ 9966.183165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9971.481893] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 9979.835903] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9986.305250] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9994.515026] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10002.259632] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [10012.133467] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10160.099744] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10160.106647] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10160.121039] Lustre: Skipped 3 previous similar messages [10160.128976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10160.133756] Lustre: Skipped 5 previous similar messages [10162.627133] Lustre: server umount lustre-OST0000 complete [10172.764283] Lustre: server umount lustre-MDT0000 complete [10175.467858] LustreError: 246130:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10175.481058] LustreError: 246130:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 2 previous similar messages [10176.356874] LustreError: 245385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787144546 with bad export cookie 255252056052019892 [10176.374903] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10176.379657] LustreError: 245385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10176.789392] Lustre: server umount lustre-MDT0001 complete [10281.012229] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [10283.921748] Key type lgssc unregistered [10284.193704] LNet: 249025:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10284.199644] LNetError: 249025:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10284.213226] LNet: Removed LNI 192.168.201.158@tcp [10285.013333] Key type .llcrypt unregistered [10285.016173] Key type ._llcrypt unregistered [10311.833593] Key type ._llcrypt registered [10311.839279] Key type .llcrypt registered [10311.971918] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [10329.860676] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [10330.972931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10331.034604] alg: No test for adler32 (adler32-zlib) [10332.263529] Lustre: Lustre: Build Version: 2.17.57_11_g9d7c9d8 [10332.480744] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [10334.152315] Key type lgssc registered [10335.167679] Lustre: Echo OBD driver; http://www.lustre.org/ [10391.537506] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [10402.856026] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10402.875821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10404.241083] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10404.299912] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10404.374582] Lustre: lustre-MDT0000: new disk, initializing [10404.470983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10404.498533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10411.337193] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [10422.662232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10422.782070] Lustre: 253094:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10422.832929] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10422.837174] Lustre: Skipped 1 previous similar message [10422.906742] Lustre: lustre-MDT0001: new disk, initializing [10422.999287] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10423.053741] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10423.071660] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10427.977602] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [10432.841518] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10440.052857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10440.523562] Lustre: lustre-OST0000: new disk, initializing [10440.527104] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10440.537829] Lustre: 254726:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10440.653536] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10447.446269] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [10450.034756] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10450.053623] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10450.150557] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10458.250200] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10458.420309] Lustre: lustre-OST0001: new disk, initializing [10458.424570] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10458.431510] Lustre: 255599:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10458.528409] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10464.868657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10464.879707] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10464.971486] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10465.249811] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [10476.040710] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10483.149088] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10494.958194] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10494.968750] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10494.985102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10495.970489] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10495.973272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10496.007541] Lustre: Skipped 2 previous similar messages [10496.023240] Lustre: Skipped 3 previous similar messages [10501.073141] Lustre: server umount lustre-MDT0000 complete [10505.824225] LustreError: 253087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787144875 with bad export cookie 5387930840756645894 [10505.831104] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10505.843143] LustreError: 253087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10506.213177] LustreError: 253104:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10506.218026] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10506.242853] LustreError: 253104:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 6 previous similar messages [10506.418401] Lustre: server umount lustre-MDT0001 complete [10512.926762] Lustre: server umount lustre-OST0000 complete [10517.042427] Lustre: server umount lustre-OST0001 complete [10530.620571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10539.372569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10547.252458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10556.190893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10575.766597] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [10588.102753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10588.145623] Lustre: lustre-MDT0000: reset Object Index mappings [10588.749836] LustreError: 260610:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10588.800391] LustreError: 260610:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 2 previous similar messages [10588.868132] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10593.690631] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [10594.289245] LustreError: 260611:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10598.370310] LustreError: 260610:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10602.255281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10602.274937] Lustre: lustre-MDT0001: reset Object Index mappings [10602.859806] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10607.211771] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [10609.907770] Lustre: 261752:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10616.867104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10623.790262] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [10627.498619] LustreError: 262107:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10632.972803] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10633.222935] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10633.225988] Lustre: Skipped 1 previous similar message [10634.310868] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10634.319972] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10639.905544] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [10647.947589] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10651.692103] Lustre: 263624:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10658.368882] Lustre: server umount lustre-MDT0000 complete [10659.297865] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10659.304879] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10659.315330] Lustre: Skipped 1 previous similar message [10659.329317] LustreError: 260611:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10659.343052] LustreError: 260611:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 1 previous similar message [10662.207235] LustreError: 260591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787145032 with bad export cookie 5387930840756652684 [10662.213200] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10662.225695] LustreError: 260591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10662.524614] Lustre: server umount lustre-MDT0001 complete [10676.764023] Lustre: server umount lustre-OST0000 complete [10690.927745] Lustre: server umount lustre-OST0001 complete [10711.727564] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 09:11:19 (1787145079) [10727.766950] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [10738.875211] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [10781.127099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10781.457874] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10781.504168] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10781.567615] Lustre: lustre-MDT0000: new disk, initializing [10781.647397] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10781.667860] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10786.578931] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [10799.321673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10799.509827] Lustre: 267882:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10799.594275] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10799.602138] Lustre: Skipped 1 previous similar message [10799.726792] Lustre: lustre-MDT0001: new disk, initializing [10800.003310] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10800.058767] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10806.556729] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [10819.274369] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10826.673184] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10834.942617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10835.417342] Lustre: lustre-OST0000: new disk, initializing [10835.427224] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10835.442828] Lustre: 269853:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10837.185920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10837.202985] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10837.402155] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10842.135476] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [10852.630522] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10858.721780] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10859.013507] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10863.201346] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10863.457290] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10868.193923] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10868.194915] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10868.203236] LustreError: Skipped 1 previous similar message [10868.206402] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10868.221124] Lustre: Skipped 4 previous similar messages [10868.235586] Lustre: Skipped 1 previous similar message [10871.870184] Lustre: server umount lustre-OST0000 complete [10876.910693] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10876.924216] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10876.966987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10883.083386] Lustre: server umount lustre-MDT0000 complete [10883.555094] LustreError: 267893:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10883.569179] LustreError: 267893:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 4 previous similar messages [10888.253601] LustreError: 267874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787145258 with bad export cookie 5387930840756654777 [10888.259436] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10888.267064] LustreError: 267874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10888.826423] Lustre: server umount lustre-MDT0001 complete [10911.307628] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [10923.633982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10924.169500] LustreError: 272788:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10924.203801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10924.208701] Lustre: Skipped 2 previous similar messages [10931.564657] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [10942.536294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10948.161798] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [10951.151262] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10958.261817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10965.809989] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [10975.657889] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10975.893871] Lustre: lustre-OST0001: new disk, initializing [10975.897414] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10975.907734] Lustre: 275099:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10978.004693] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10978.012674] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10978.180151] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10983.093930] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [10992.966545] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10997.659697] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11003.369456] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [11003.724668] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [11014.117140] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11014.123391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11014.151723] Lustre: Skipped 3 previous similar messages [11014.167806] Lustre: Skipped 5 previous similar messages [11024.352069] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11024.373518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11024.389877] Lustre: Skipped 5 previous similar messages [11024.698284] Lustre: server umount lustre-MDT0000 complete [11029.111830] LustreError: 272770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787145399 with bad export cookie 5387930840756655659 [11029.128940] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11029.136900] LustreError: 272770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11029.477802] LustreError: 275062:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11029.515878] LustreError: 275062:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 4 previous similar messages [11029.725595] Lustre: server umount lustre-MDT0001 complete [11037.055057] Lustre: server umount lustre-OST0000 complete [11043.219499] Lustre: server umount lustre-OST0001 complete [11066.943965] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [11083.365817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11084.023266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11084.029717] Lustre: Skipped 3 previous similar messages [11089.414374] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [11099.697784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11105.211541] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [11108.829931] Lustre: 279295:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11117.008262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11125.578301] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [11128.955707] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [11135.346807] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11140.628845] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [11143.164859] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [11153.692412] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11158.973747] Lustre: 281175:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11166.290654] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [11166.384585] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [11180.577727] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 09:19:08 (1787145548) [11202.022290] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11202.037506] Lustre: Skipped 5 previous similar messages [11202.037596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11202.052810] Lustre: Skipped 5 previous similar messages [11203.976493] Lustre: server umount lustre-MDT0000 complete [11207.161802] LustreError: 278151:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11207.206457] LustreError: 278151:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 15 previous similar messages [11208.457485] LustreError: 278135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787145578 with bad export cookie 5387930840756662232 [11208.461093] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11208.468767] LustreError: 278135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11214.934970] Lustre: server umount lustre-MDT0001 complete [11225.972246] Lustre: server umount lustre-OST0000 complete [11232.647422] Lustre: server umount lustre-OST0001 complete [11241.435892] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [11256.665949] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [11310.644969] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [11321.961027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11322.175298] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11322.194636] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11322.304725] Lustre: lustre-MDT0000: new disk, initializing [11322.417577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11327.355894] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [11337.275078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11337.360732] Lustre: 285438:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11337.636381] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11342.179093] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [11355.962427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11356.660627] Lustre: 287073:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11356.790759] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11356.793714] Lustre: Skipped 5 previous similar messages [11358.128782] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11365.692652] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [11378.719684] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11378.904530] Lustre: 287945:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11381.254128] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11381.265440] Lustre: Skipped 1 previous similar message [11381.444250] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11387.375213] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [11398.391563] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11402.275474] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11402.282371] Lustre: Skipped 1 previous similar message [11417.011957] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 09:23:06 (1787145786) [11419.618805] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11419.631718] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11419.652646] Lustre: Skipped 2 previous similar messages [11419.662792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11419.670667] Lustre: Skipped 4 previous similar messages [11421.840952] Lustre: server umount lustre-MDT0000 complete [11426.434726] LustreError: 285429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787145796 with bad export cookie 5387930840756669890 [11426.446532] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11426.450889] LustreError: 285429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11426.964401] Lustre: server umount lustre-MDT0001 complete [11441.752686] Lustre: server umount lustre-OST0000 complete [11443.300715] Lustre: 250602:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787145797/real 1787145797] req@ffff8d082aa93b80 x1873957043446656/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787145813 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11446.948152] Lustre: server umount lustre-OST0001 complete [11455.952850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11460.952740] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [11466.722713] LustreError: 290189:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11466.740327] LustreError: 290189:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 10 previous similar messages [11470.230631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11475.377096] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [11483.827820] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11489.391637] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11497.310184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11504.196512] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [11512.188045] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11534.158079] loop1: detected capacity change from 0 to 204800000 [11534.655017] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11534.690497] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11534.713525] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11541.922471] loop1: detected capacity change from 0 to 204800000 [11548.191274] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11548.254170] LustreError: 290178:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs8592: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11548.262720] LustreError: 290178:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs8592 [11548.278667] LustreError: 290178:0:(mgs_handler.c:517:mgs_target_reg()) fs8592-MDT0001: Error checking target index: rc = -13 [11548.285199] LustreError: fs8592-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11548.291235] LustreError: 293910:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11548.294618] LustreError: 293910:0:(tgt_mount.c:2129:server_put_super()) no obd fs8592-MDT0001 [11548.299515] LustreError: 293910:0:(tgt_mount.c:116:server_deregister_mount()) fs8592-MDT0001 not registered [11548.329293] Lustre: server umount fs8592-MDT0001 complete [11548.333132] LustreError: 293910:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11559.427554] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11559.581081] Lustre: Setting parameter fs8592-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs8592-MDT0001 [11559.626348] Lustre: srv-fs8592-MDT0001: No data found on store. Initialize space. [11559.628889] Lustre: Skipped 4 previous similar messages [11559.703410] Lustre: fs8592-MDT0001: new disk, initializing [11559.707379] Lustre: Skipped 3 previous similar messages [11559.903483] Lustre: srv-fs8592-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11559.922534] Lustre: 294378:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs8592-MDT0001-mdtlov: can't create new llog: rc = -115 [11559.940348] LustreError: 294378:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs8592-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11566.324939] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [11574.219450] Lustre: server umount fs8592-MDT0001 complete [11581.922480] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11593.555259] loop1: detected capacity change from 0 to 204800000 [11600.135444] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11600.692746] Lustre: 295946:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs8592-MDT0001-mdtlov: can't create new llog: rc = -115 [11600.699524] LustreError: 295946:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs8592-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11605.435296] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [11611.337600] Lustre: server umount fs8592-MDT0001 complete [11622.891888] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11622.897828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11622.911948] Lustre: Skipped 6 previous similar messages [11622.934036] Lustre: Skipped 2 previous similar messages [11636.191744] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11636.494388] Lustre: server umount lustre-MDT0000 complete [11647.216358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11647.415702] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11653.110175] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11653.256433] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [11662.116593] loop1: detected capacity change from 0 to 204800000 [11668.726500] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11669.196384] Lustre: 298394:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs8592-MDT0001-mdtlov: can't create new llog: rc = -115 [11669.212246] LustreError: 298394:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs8592-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11674.669474] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [11680.903169] Lustre: server umount fs8592-MDT0001 complete [11709.427375] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11709.456270] LustreError: Skipped 1 previous similar message [11712.287523] Lustre: server umount lustre-OST0000 complete [11723.471978] Lustre: server umount lustre-MDT0000 complete [11728.435738] LustreError: 290169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787146098 with bad export cookie 5387930840756672571 [11728.446447] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11728.452362] LustreError: 290169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11729.004369] Lustre: server umount lustre-MDT0001 complete [11740.442904] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [11743.531602] Key type lgssc unregistered [11743.981637] LNet: 300742:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11743.989772] LNetError: 300742:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11744.014528] LNet: Removed LNI 192.168.201.158@tcp [11745.276202] Key type .llcrypt unregistered [11745.277950] Key type ._llcrypt unregistered [11767.380599] Key type ._llcrypt registered [11767.382447] Key type .llcrypt registered [11767.484287] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [11768.604426] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11768.871646] alg: No test for adler32 (adler32-zlib) [11770.102229] Lustre: Lustre: Build Version: 2.17.57_11_g9d7c9d8 [11770.359197] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [11772.032484] Key type lgssc registered [11773.084987] Lustre: Echo OBD driver; http://www.lustre.org/ [11780.786380] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11780.812444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11782.403809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11787.169445] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [11792.868499] LustreError: 302027:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11792.914173] LustreError: 302027:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 2 previous similar messages [11793.899653] LustreError: 302028:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11798.100984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11798.574065] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11805.067649] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [11813.295665] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11819.449273] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11827.539459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11828.049478] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11829.105989] LustreError: 303859:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11834.368110] LustreError: 303932:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11835.071808] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [11839.466257] LustreError: 303858:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11843.143614] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11848.698396] LustreError: 303858:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11848.708109] LustreError: 303858:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 4 previous similar messages [11865.756545] loop1: detected capacity change from 0 to 204800000 [11866.027150] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11866.040276] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11866.060483] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11866.957173] LustreError: 303932:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11866.989250] LustreError: 303932:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 9 previous similar messages [11872.428413] loop1: detected capacity change from 0 to 204800000 [11877.921134] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11878.112830] Lustre: fs26950-OST0000: new disk, initializing [11878.117779] Lustre: srv-fs26950-OST0000: No data found on store. Initialize space. [11878.124798] Lustre: 305764:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11878.208102] Lustre: fs26950-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11886.013812] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [11894.870734] Lustre: server umount fs26950-OST0000 complete [11899.905149] LustreError: 303858:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11899.939225] LustreError: 303858:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 19 previous similar messages [11909.891397] loop1: detected capacity change from 0 to 204800000 [11910.121192] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11915.361361] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11915.606367] Lustre: fs26950-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11922.918427] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [11925.489792] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11932.860774] loop2: detected capacity change from 0 to 204800000 [11933.285937] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11933.305602] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11933.316781] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11939.724606] loop2: detected capacity change from 0 to 204800000 [11946.116776] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11946.321048] LustreError: fs26950-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11946.343679] LustreError: fs26950-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11946.355893] LustreError: 308803:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11946.365798] LustreError: 308803:0:(tgt_mount.c:2129:server_put_super()) no obd fs26950-OST0001 [11946.372799] LustreError: 308803:0:(tgt_mount.c:116:server_deregister_mount()) fs26950-OST0001 not registered [11946.433811] Lustre: server umount fs26950-OST0001 complete [11946.440878] LustreError: 308803:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11958.250354] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11958.342747] Lustre: fs26950-OST0001: new disk, initializing [11958.346383] Lustre: srv-fs26950-OST0001: No data found on store. Initialize space. [11958.351492] Lustre: 309354:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11958.424060] Lustre: fs26950-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11964.237039] LustreError: 303932:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11964.275584] LustreError: 303932:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 36 previous similar messages [11965.991718] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [11976.552657] Lustre: server umount fs26950-OST0000 complete [11986.409396] Lustre: server umount fs26950-OST0001 complete [11999.449488] loop1: detected capacity change from 0 to 204800000 [12005.227681] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12005.401433] Lustre: fs26950-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12011.638346] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [12018.667708] loop2: detected capacity change from 0 to 204800000 [12024.337346] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12031.030928] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [12041.145391] Lustre: server umount fs26950-OST0000 complete [12050.561848] Lustre: server umount fs26950-OST0001 complete [12063.724071] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12063.725346] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [12063.741125] Lustre: Skipped 1 previous similar message [12063.746382] LustreError: Skipped 1 previous similar message [12063.747860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12068.843135] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12068.847414] Lustre: Skipped 1 previous similar message [12069.594463] Lustre: server umount lustre-OST0000 complete [12073.962841] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12073.965567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12073.970865] Lustre: Skipped 1 previous similar message [12073.978685] Lustre: Skipped 2 previous similar messages [12079.072678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12079.083753] Lustre: Skipped 1 previous similar message [12079.510700] Lustre: server umount lustre-MDT0000 complete [12083.774758] LustreError: 302006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787146453 with bad export cookie 8377948679150578862 [12083.784935] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12083.789097] LustreError: 302006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12084.161737] Lustre: server umount lustre-MDT0001 complete [12097.037564] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [12099.243536] Key type lgssc unregistered [12099.530537] LNet: 315281:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12099.542681] LNetError: 315281:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12099.569174] LNet: Removed LNI 192.168.201.158@tcp [12100.505126] Key type .llcrypt unregistered [12100.507714] Key type ._llcrypt unregistered [12134.744435] Key type ._llcrypt registered [12134.746428] Key type .llcrypt registered [12134.836412] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [12153.703278] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [12155.112537] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12155.169616] alg: No test for adler32 (adler32-zlib) [12156.203988] Lustre: Lustre: Build Version: 2.17.57_11_g9d7c9d8 [12156.442635] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [12158.071283] Key type lgssc registered [12159.123605] Lustre: Echo OBD driver; http://www.lustre.org/ [12200.241550] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12200.298373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12201.708955] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12201.780246] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12201.865590] Lustre: lustre-MDT0000: new disk, initializing [12202.031820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12202.061257] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12206.040307] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [12219.623040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12219.824480] Lustre: 319147:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12219.952268] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12219.971215] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [12219.973145] Lustre: Skipped 1 previous similar message [12220.136077] Lustre: lustre-MDT0001: new disk, initializing [12220.395082] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12220.450127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12220.475341] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12226.345526] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [12237.308906] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12244.350436] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12252.814512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12253.210744] Lustre: lustre-OST0000: new disk, initializing [12253.219440] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12253.231505] Lustre: 321116:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12253.320166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12261.432054] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [12262.558849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12262.579531] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12262.725873] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12271.882522] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12274.086561] Lustre: Mounted lustre-client - version 2.17.57_11_g9d7c9d8 [12275.171282] Lustre: Unmounted lustre-client [12288.482974] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12288.485739] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12288.495314] LustreError: Skipped 1 previous similar message [12288.511103] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12288.519213] Lustre: Skipped 1 previous similar message [12294.627161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12294.636557] Lustre: Skipped 4 previous similar messages [12299.751665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12299.762454] Lustre: Skipped 1 previous similar message [12300.255226] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12300.596145] Lustre: server umount lustre-OST0000 complete [12307.427674] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12307.435327] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12307.458914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12309.986333] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12312.532309] Lustre: server umount lustre-MDT0000 complete [12317.156758] LustreError: 319137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787146687 with bad export cookie 7995397704733843858 [12317.162396] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12317.168612] LustreError: 319137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12317.501512] Lustre: server umount lustre-MDT0001 complete [12325.472136] LNet: 1 local NIs in recovery (showing 1): 192.168.201.158@tcp [12330.760768] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [12333.942618] Key type lgssc unregistered [12334.352563] LNet: 323286:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12334.364404] LNetError: 323286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12404.010637] LNet: Removed LNI 192.168.201.158@tcp [12404.951669] Key type .llcrypt unregistered [12404.965729] Key type ._llcrypt unregistered [12425.032536] Key type ._llcrypt registered [12425.037486] Key type .llcrypt registered [12425.144470] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [12445.146878] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [12447.618334] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12447.682226] alg: No test for adler32 (adler32-zlib) [12448.781204] Lustre: Lustre: Build Version: 2.17.57_11_g9d7c9d8 [12449.104606] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [12450.856687] Key type lgssc registered [12451.839670] Lustre: Echo OBD driver; http://www.lustre.org/ [12486.772086] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 09:40:55 (1787146855) [12494.432541] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12494.521290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12495.863242] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12495.903554] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12496.006510] Lustre: lustre-MDT0000: new disk, initializing [12496.079169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12496.111910] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12500.924189] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [12513.803192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12514.031437] Lustre: 326511:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12514.163572] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12514.170926] Lustre: Skipped 1 previous similar message [12514.341104] Lustre: lustre-MDT0001: new disk, initializing [12514.510256] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12514.578452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12514.613961] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12520.426663] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [12530.171316] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12536.423906] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12543.257185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12543.446289] Lustre: lustre-OST0000: new disk, initializing [12543.448557] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12543.454601] Lustre: 328479:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12543.563379] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12550.531255] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [12552.307704] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12552.339171] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12552.512422] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12561.685823] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12583.400734] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12583.402135] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12583.415807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12583.427088] Lustre: Skipped 1 previous similar message [12586.254952] Lustre: server umount lustre-OST0000 complete [12591.584266] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12591.605290] LustreError: Skipped 1 previous similar message [12591.613451] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12591.635162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12591.640390] Lustre: Skipped 1 previous similar message [12593.635113] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12593.635208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12593.665921] Lustre: Skipped 1 previous similar message [12598.753270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12603.885247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12603.890950] Lustre: Skipped 2 previous similar messages [12604.383134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12604.705646] Lustre: server umount lustre-MDT0000 complete [12608.519523] LustreError: 326503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787146978 with bad export cookie 13159598376296025041 [12608.533978] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12608.869463] Lustre: server umount lustre-MDT0001 complete [12621.969545] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [12625.257886] Key type lgssc unregistered [12625.551855] LNet: 330391:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12625.568128] LNetError: 330391:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12625.594367] LNet: Removed LNI 192.168.201.158@tcp [12626.500858] Key type .llcrypt unregistered [12626.503501] Key type ._llcrypt unregistered [12651.058355] Key type ._llcrypt registered [12651.060055] Key type .llcrypt registered [12651.169292] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [12652.376655] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12652.396212] alg: No test for adler32 (adler32-zlib) [12653.441835] Lustre: Lustre: Build Version: 2.17.57_11_g9d7c9d8 [12653.801825] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [12655.527190] Key type lgssc registered [12657.064900] Lustre: Echo OBD driver; http://www.lustre.org/ [12665.492344] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12665.512548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12667.230172] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12672.163579] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [12677.605241] LustreError: 331665:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12677.620832] LustreError: 331665:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 2 previous similar messages [12682.667592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12682.720370] LustreError: 331666:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12682.998086] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12688.641803] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [12698.215638] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12707.098598] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12715.600915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12715.887427] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12723.233968] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [12731.746571] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12746.274924] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12747.809630] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12752.868941] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12752.870203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12752.889401] Lustre: Skipped 1 previous similar message [12752.913399] Lustre: Skipped 2 previous similar messages [12758.004867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12758.007494] Lustre: Skipped 1 previous similar message [12758.345481] LustreError: 331661:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12760.265388] Lustre: server umount lustre-MDT0000 complete [12763.115937] LustreError: 331662:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12763.144639] LustreError: 331662:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 2 previous similar messages [12765.337146] LustreError: 331647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787147135 with bad export cookie 8397826856993547413 [12765.346876] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12768.224991] LustreError: 331661:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12768.230909] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12768.264445] LustreError: 331661:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 1 previous similar message [12768.310396] Lustre: Skipped 1 previous similar message [12768.336696] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12771.960442] Lustre: server umount lustre-MDT0001 complete [12776.747766] Lustre: server umount lustre-OST0000 complete [12786.961843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12792.012200] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [12795.916962] Lustre: 335855:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.158@tcp [12798.590631] Lustre: server umount MGS complete [12810.128525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12810.802032] LustreError: 336426:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12810.829480] LustreError: 336426:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 2 previous similar messages [12810.928246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12816.131630] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [12827.249395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12827.705392] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12832.657489] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [12842.075276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12842.493884] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12849.066227] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [12857.265143] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12859.094815] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12866.826814] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12868.721358] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12874.728222] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [12874.736551] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12874.754584] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12874.766594] Lustre: Skipped 2 previous similar messages [12876.623661] Lustre: server umount lustre-OST0000 complete [12881.377942] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12881.393907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12881.397300] Lustre: Skipped 2 previous similar messages [12881.421603] Lustre: Skipped 3 previous similar messages [12887.015665] Lustre: server umount lustre-MDT0000 complete [12890.650949] LustreError: 336408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787147260 with bad export cookie 8397826856993548253 [12890.669413] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12890.677912] LustreError: 336408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [12891.142720] Lustre: server umount lustre-MDT0001 complete [12902.615676] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [12905.018240] Key type lgssc unregistered [12905.296828] LNet: 339864:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12905.322257] LNetError: 339864:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12905.347174] LNet: Removed LNI 192.168.201.158@tcp [12906.226869] Key type .llcrypt unregistered [12906.229416] Key type ._llcrypt unregistered [12927.602163] Key type ._llcrypt registered [12927.605459] Key type .llcrypt registered [12927.776663] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [12945.928453] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [12947.607650] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12947.729600] alg: No test for adler32 (adler32-zlib) [12949.227059] Lustre: Lustre: Build Version: 2.17.57_11_g9d7c9d8 [12949.445166] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [12951.087164] Key type lgssc registered [12952.098557] Lustre: Echo OBD driver; http://www.lustre.org/ [12991.740908] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12991.775659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12993.108227] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12993.138619] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12993.256273] Lustre: lustre-MDT0000: new disk, initializing [12993.422658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12993.446082] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12997.781091] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [13009.599986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13009.749198] Lustre: 343038:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13009.807416] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13009.812955] Lustre: Skipped 1 previous similar message [13009.873591] Lustre: lustre-MDT0001: new disk, initializing [13009.967644] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13010.011763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13010.026919] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13015.377499] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [13025.333512] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13031.070507] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13038.165596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13038.445297] Lustre: lustre-OST0000: new disk, initializing [13038.456150] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13038.475851] Lustre: 345008:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13038.614065] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13044.676310] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [13048.423749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13048.437030] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13048.576212] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13055.132353] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13060.187463] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13060.405512] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13065.219227] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13065.398933] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13069.287765] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13069.292392] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13069.298202] LustreError: Skipped 1 previous similar message [13069.323229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13069.335798] Lustre: Skipped 1 previous similar message [13073.857300] Lustre: server umount lustre-OST0000 complete [13079.527658] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13079.535644] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13079.544060] Lustre: Skipped 2 previous similar messages [13079.554928] Lustre: Skipped 1 previous similar message [13085.137895] Lustre: server umount lustre-MDT0000 complete [13089.760815] LustreError: 343046:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13089.815449] LustreError: 343046:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 4 previous similar messages [13090.794213] LustreError: 344214:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13090.835996] LustreError: 344214:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 1 previous similar message [13092.708201] LustreError: 343029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787147462 with bad export cookie 4717960546632225385 [13092.710084] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13092.737584] LustreError: 343029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13093.645476] Lustre: server umount lustre-MDT0001 complete [13109.083942] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 09:51:17 (1787147477) [13129.880463] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [13140.424045] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [13174.701516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13175.034943] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13175.065398] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13175.126970] Lustre: lustre-MDT0000: new disk, initializing [13175.211506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13175.223485] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13179.633351] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [13190.164288] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13190.336235] Lustre: 349771:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13190.443382] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13190.455079] Lustre: Skipped 1 previous similar message [13190.583661] Lustre: lustre-MDT0001: new disk, initializing [13190.748729] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13190.821732] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13190.842544] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13195.409772] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [13204.766424] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13210.456615] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13217.872238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13218.227420] Lustre: lustre-OST0000: new disk, initializing [13218.231587] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13218.236885] Lustre: 351737:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13218.332783] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13224.767978] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [13226.631408] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13226.652923] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13226.829266] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13234.049938] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13267.936980] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13267.939844] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13267.943074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13267.946672] Lustre: Skipped 1 previous similar message [13267.959546] LustreError: Skipped 1 previous similar message [13269.339335] Lustre: server umount lustre-OST0000 complete [13277.670728] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13277.681408] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13277.709088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13277.720372] Lustre: Skipped 1 previous similar message [13279.574183] Lustre: server umount lustre-MDT0000 complete [13282.924702] LustreError: 350574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787147652 with bad export cookie 4717960546632226260 [13282.929458] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13282.934782] LustreError: 350574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13283.226455] Lustre: server umount lustre-MDT0001 complete [13294.463284] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [13296.742842] Key type lgssc unregistered [13296.997512] LNet: 353894:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13297.002502] LNetError: 353894:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13297.024769] LNet: Removed LNI 192.168.201.158@tcp [13297.757563] Key type .llcrypt unregistered [13297.759805] Key type ._llcrypt unregistered [13403.197630] Key type ._llcrypt registered [13403.199694] Key type .llcrypt registered [13403.271329] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [13403.576439] Key type .llcrypt unregistered [13403.579751] Key type ._llcrypt unregistered [13486.180140] Key type ._llcrypt registered [13486.181860] Key type .llcrypt registered [13486.292687] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [13486.776327] Key type .llcrypt unregistered [13486.778297] Key type ._llcrypt unregistered [13520.912890] Key type ._llcrypt registered [13520.916974] Key type .llcrypt registered [13521.114316] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [13521.427696] Key type .llcrypt unregistered [13521.431565] Key type ._llcrypt unregistered [13553.124580] Key type ._llcrypt registered [13553.127202] Key type .llcrypt registered [13553.243794] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [13553.573640] Key type .llcrypt unregistered [13553.577764] Key type ._llcrypt unregistered [13636.702480] Key type ._llcrypt registered [13636.708466] Key type .llcrypt registered [13636.844216] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [13637.251190] Key type .llcrypt unregistered [13637.263603] Key type ._llcrypt unregistered [13666.411263] Key type ._llcrypt registered [13666.413601] Key type .llcrypt registered [13666.516780] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [13666.803049] Key type .llcrypt unregistered [13666.804487] Key type ._llcrypt unregistered [13706.308264] Key type ._llcrypt registered [13706.316654] Key type .llcrypt registered [13706.483506] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [13706.944445] Key type .llcrypt unregistered [13706.952061] Key type ._llcrypt unregistered [13746.774677] Key type ._llcrypt registered [13746.777139] Key type .llcrypt registered [13746.868801] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [13747.236557] Key type .llcrypt unregistered [13747.240473] Key type ._llcrypt unregistered [13787.683077] Key type ._llcrypt registered [13787.684536] Key type .llcrypt registered [13787.851700] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [13788.214915] Key type .llcrypt unregistered [13788.219925] Key type ._llcrypt unregistered [13824.626887] Key type ._llcrypt registered [13824.633741] Key type .llcrypt registered [13824.740967] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [13825.131618] Key type .llcrypt unregistered [13825.133521] Key type ._llcrypt unregistered [13893.858421] Key type ._llcrypt registered [13893.863262] Key type .llcrypt registered [13893.975692] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [13894.411315] Key type .llcrypt unregistered [13894.415373] Key type ._llcrypt unregistered [13930.998382] Key type ._llcrypt registered [13931.002372] Key type .llcrypt registered [13931.228904] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [13931.683710] Key type .llcrypt unregistered [13931.685354] Key type ._llcrypt unregistered [13964.990962] Key type ._llcrypt registered [13964.993884] Key type .llcrypt registered [13965.125619] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [13965.453534] Key type .llcrypt unregistered [13965.459905] Key type ._llcrypt unregistered [14025.014101] Key type ._llcrypt registered [14025.019751] Key type .llcrypt registered [14025.201349] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [14042.332739] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [14044.301266] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14044.357841] alg: No test for adler32 (adler32-zlib) [14045.710592] Lustre: Lustre: Build Version: 2.17.57_11_g9d7c9d8 [14046.009074] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [14047.767827] Key type lgssc registered [14049.935776] Lustre: Echo OBD driver; http://www.lustre.org/ [14095.598581] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14095.646275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14096.906859] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [14096.926088] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [14097.005958] Lustre: tst250fs-MDT0000: new disk, initializing [14097.079087] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14097.092505] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14102.663890] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [14113.739066] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14113.853439] Lustre: 374295:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14113.916719] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [14113.920613] Lustre: Skipped 1 previous similar message [14114.007934] Lustre: tst250fs-MDT0001: new disk, initializing [14114.101079] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14114.178057] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14114.189725] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14119.920216] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [14124.996304] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [14132.083109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14132.527871] Lustre: tst250fs-OST0000: new disk, initializing [14132.531547] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [14132.556618] Lustre: 375928:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14132.747562] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14134.588590] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14134.613679] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14134.803338] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14141.014591] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [14152.139215] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14152.321400] Lustre: tst250fs-OST0001: new disk, initializing [14152.325685] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [14152.333181] Lustre: 376800:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14152.447296] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [14159.725803] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [14161.551656] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [14161.563859] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [14161.675796] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [14165.870920] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [14168.613056] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [14168.620301] Lustre: Skipped 2 previous similar messages [14191.076913] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14191.084205] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14191.099865] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14192.609382] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14192.612826] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14192.620230] Lustre: Skipped 1 previous similar message [14192.645958] Lustre: Skipped 1 previous similar message [14194.511037] Lustre: server umount tst250fs-MDT0000 complete [14198.372889] LustreError: 374288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787148568 with bad export cookie 13771902821066900338 [14198.376749] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14198.382886] LustreError: 374288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14198.799417] Lustre: server umount tst250fs-MDT0001 complete [14213.703859] Lustre: server umount tst250fs-OST0000 complete [14227.600829] Lustre: server umount tst250fs-OST0001 complete [14247.863473] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [14258.753722] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [14293.565291] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 10:11:02 (1787148662) [14295.442572] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [14316.569470] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [14334.846565] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [14370.090068] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14054 sec ========= 10:12:18 (1787148738) [14372.476574] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 10:12:20 (1787148740) === [14376.228357] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 10:12:24 (1787148744) === [14425.983311] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [14428.611041] Key type lgssc unregistered [14428.852836] LNet: 385956:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14428.868287] LNetError: 385956:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14428.882380] LNet: Removed LNI 192.168.201.158@tcp [14429.798413] Key type .llcrypt unregistered [14429.800502] Key type ._llcrypt unregistered