[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 501865073 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003377] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005020] kvm-guest: setup PV IPIs [ 0.007785] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009016] pid_max: default: 32768 minimum: 301 [ 0.010149] LSM: Security Framework initializing [ 0.011068] Yama: becoming mindful. [ 0.012053] SELinux: Initializing. [ 0.013085] *** VALIDATE selinux *** [ 0.021879] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026600] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028043] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030090] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032033] *** VALIDATE tmpfs *** [ 0.034105] *** VALIDATE proc *** [ 0.035333] *** VALIDATE cgroup *** [ 0.037009] *** VALIDATE cgroup2 *** [ 0.038350] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039172] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041032] Spectre V2 : User space: Vulnerable [ 0.042013] Speculative Store Bypass: Vulnerable [ 0.045257] debug: unmapping init [mem 0xffffffffb0859000-0xffffffffb0860fff] [ 0.047185] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048692] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049026] ... version: 2 [ 0.050019] ... bit width: 48 [ 0.051021] ... generic registers: 4 [ 0.052028] ... value mask: 0000ffffffffffff [ 0.053018] ... max period: 00007fffffffffff [ 0.054019] ... fixed-purpose events: 3 [ 0.055019] ... event mask: 000000070000000f [ 0.056335] rcu: Hierarchical SRCU implementation. [ 0.058568] smp: Bringing up secondary CPUs ... [ 0.059539] x86: Booting SMP configuration: [ 0.060036] .... node #0, CPUs: #1 #2 #3 [ 0.063349] smp: Brought up 1 node, 4 CPUs [ 0.065019] smpboot: Max logical packages: 1 [ 0.066022] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.143946] node 0 deferred pages initialised in 73ms [ 0.147162] devtmpfs: initialized [ 0.149288] x86/mm: Memory block size: 128MB [ 0.153873] gcov: version magic: 0x41383552 [ 0.157306] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.159084] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.162352] pinctrl core: initialized pinctrl subsystem [ 0.164198] [ 0.164849] ************************************************************* [ 0.168013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.170011] ** ** [ 0.173014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.175012] ** ** [ 0.178014] ** This means that this kernel is built to expose internal ** [ 0.180011] ** IOMMU data structures, which may compromise security on ** [ 0.183024] ** your system. ** [ 0.185013] ** ** [ 0.188012] ** If you see this message and you are not debugging the ** [ 0.190012] ** kernel, report this immediately to your vendor! ** [ 0.192013] ** ** [ 0.195016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.197011] ************************************************************* [ 0.200917] NET: Registered protocol family 16 [ 0.202412] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.205059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.208067] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.211451] cpuidle: using governor menu [ 0.212891] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.215607] PCI: Using configuration type 1 for base access [ 0.217121] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.229060] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.230083] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.234142] cryptd: max_cpu_qlen set to 1000 [ 0.238264] ACPI: Added _OSI(Module Device) [ 0.240015] ACPI: Added _OSI(Processor Device) [ 0.242015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.243011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.250319] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.258567] ACPI: Interpreter enabled [ 0.260062] ACPI: PM: (supports S0 S3 S4 S5) [ 0.262016] ACPI: Using IOAPIC for interrupt routing [ 0.263095] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.267236] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.278344] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.280044] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.283020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.288071] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.294912] acpiphp: Slot [2] registered [ 0.297141] acpiphp: Slot [5] registered [ 0.299089] acpiphp: Slot [6] registered [ 0.301161] acpiphp: Slot [7] registered [ 0.303153] acpiphp: Slot [8] registered [ 0.304198] acpiphp: Slot [9] registered [ 0.306139] acpiphp: Slot [10] registered [ 0.307112] acpiphp: Slot [3] registered [ 0.308096] acpiphp: Slot [4] registered [ 0.310102] acpiphp: Slot [11] registered [ 0.311128] acpiphp: Slot [12] registered [ 0.313116] acpiphp: Slot [13] registered [ 0.314111] acpiphp: Slot [14] registered [ 0.315067] acpiphp: Slot [15] registered [ 0.316068] acpiphp: Slot [16] registered [ 0.318122] acpiphp: Slot [17] registered [ 0.319098] acpiphp: Slot [18] registered [ 0.320122] acpiphp: Slot [19] registered [ 0.322115] acpiphp: Slot [20] registered [ 0.324182] acpiphp: Slot [21] registered [ 0.325104] acpiphp: Slot [22] registered [ 0.327109] acpiphp: Slot [23] registered [ 0.328118] acpiphp: Slot [24] registered [ 0.330077] acpiphp: Slot [25] registered [ 0.331054] acpiphp: Slot [26] registered [ 0.333150] acpiphp: Slot [27] registered [ 0.335140] acpiphp: Slot [28] registered [ 0.337216] acpiphp: Slot [29] registered [ 0.339115] acpiphp: Slot [30] registered [ 0.341119] acpiphp: Slot [31] registered [ 0.342064] PCI host bridge to bus 0000:00 [ 0.343022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.345024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.348026] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.351023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.352021] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.353036] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.355177] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.358055] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.360285] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.369752] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.373000] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.373021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.375019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.378020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.381566] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.383950] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.386049] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.389947] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.395016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.409017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.416016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.422129] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.432020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.443019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.467019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.482058] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.497029] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.515018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.538017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.548452] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.560015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.567019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.593020] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.604000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.615017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.623015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.644017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.657761] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.665016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.672019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.688017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.701000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.711017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.717021] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.742023] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.759388] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.762409] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.766576] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.769937] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.775274] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.782088] iommu: Default domain type: Passthrough [ 0.784469] SCSI subsystem initialized [ 0.786157] ACPI: bus type USB registered [ 0.787116] usbcore: registered new interface driver usbfs [ 0.790113] usbcore: registered new interface driver hub [ 0.792109] usbcore: registered new device driver usb [ 0.793172] pps_core: LinuxPPS API ver. 1 registered [ 0.795016] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.799064] PTP clock support registered [ 0.802084] EDAC MC: Ver: 3.0.0 [ 0.804142] PCI: Using ACPI for IRQ routing [ 0.807280] NetLabel: Initializing [ 0.809013] NetLabel: domain hash size = 128 [ 0.810009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.812082] NetLabel: unlabeled traffic allowed by default [ 0.814266] vgaarb: loaded [ 0.816264] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.818019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.824336] clocksource: Switched to clocksource kvm-clock [ 0.937124] VFS: Disk quotas dquot_6.6.0 [ 0.938885] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.941737] *** VALIDATE ramfs *** [ 0.943129] *** VALIDATE hugetlbfs *** [ 0.944504] pnp: PnP ACPI init [ 0.946792] pnp: PnP ACPI: found 6 devices [ 0.969722] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.974337] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.977254] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.980656] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.983126] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.985693] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.988584] NET: Registered protocol family 2 [ 0.991473] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.997115] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.001070] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.007381] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.011271] TCP: Hash tables configured (established 65536 bind 65536) [ 1.014398] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.017719] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.020482] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.023433] NET: Registered protocol family 1 [ 1.027829] RPC: Registered named UNIX socket transport module. [ 1.029892] RPC: Registered udp transport module. [ 1.032107] RPC: Registered tcp transport module. [ 1.033596] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.035548] NET: Registered protocol family 44 [ 1.037078] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.038962] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.041485] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.043920] PCI: CLS 0 bytes, default 64 [ 1.045598] Unpacking initramfs... [ 2.470948] debug: unmapping init [mem 0xffff98193cc54000-0xffff98193ffbffff] [ 2.474849] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.477067] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.480260] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.999502] Initialise system trusted keyrings [ 3.002154] Key type blacklist registered [ 3.004107] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.016637] zbud: loaded [ 3.020738] *** VALIDATE nfs *** [ 3.022013] *** VALIDATE nfs4 *** [ 3.023770] pstore: using deflate compression [ 3.028065] Platform Keyring initialized [ 3.128286] NET: Registered protocol family 38 [ 3.130237] Key type asymmetric registered [ 3.131893] Asymmetric key parser 'x509' registered [ 3.134137] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.137422] io scheduler mq-deadline registered [ 3.139485] io scheduler kyber registered [ 3.141241] io scheduler bfq registered [ 3.143090] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.146414] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.149482] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.152270] ACPI: Power Button [PWRF] [ 3.158082] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.165770] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.178345] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.184876] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.198228] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.224949] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.251633] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.257609] Non-volatile memory driver v1.3 [ 3.259641] Linux agpgart interface v0.103 [ 3.288198] virtio_blk virtio1: [vda] 149760 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.290650] vda: detected capacity change from 0 to 76677120 [ 3.306899] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.309760] vdb: detected capacity change from 0 to 1073741824 [ 3.323839] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.326343] vdc: detected capacity change from 0 to 2621440000 [ 3.340280] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.343309] vdd: detected capacity change from 0 to 2621440000 [ 3.359386] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.362874] vde: detected capacity change from 0 to 4294967296 [ 3.379529] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.383268] vdf: detected capacity change from 0 to 4294967296 [ 3.390189] libphy: Fixed MDIO Bus: probed [ 3.397830] usbcore: registered new interface driver usbserial_generic [ 3.400227] usbserial: USB Serial support registered for generic [ 3.402561] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.407391] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.409481] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.412560] mousedev: PS/2 mouse device common for all mice [ 3.415652] rtc_cmos 00:05: RTC can wake from S4 [ 3.418733] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.418875] rtc_cmos 00:05: registered as rtc0 [ 3.423311] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.425165] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.429134] intel_pstate: CPU model not supported [ 3.433993] hid: raw HID events driver (C) Jiri Kosina [ 3.437635] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.437739] usbcore: registered new interface driver usbhid [ 3.442773] usbhid: USB HID core driver [ 3.444263] drop_monitor: Initializing network drop monitor service [ 3.447397] Initializing XFRM netlink socket [ 3.449222] NET: Registered protocol family 10 [ 3.452127] Segment Routing with IPv6 [ 3.453537] NET: Registered protocol family 17 [ 3.455686] mpls_gso: MPLS GSO support [ 3.461901] RAS: Correctable Errors collector initialized. [ 3.464204] AVX version of gcm_enc/dec engaged. [ 3.465530] AES CTR mode by8 optimization enabled [ 3.538074] sched_clock: Marking stable (3538033950, 0)->(4498433143, -960399193) [ 3.541779] registered taskstats version 1 [ 3.544596] Loading compiled-in X.509 certificates [ 3.546191] zswap: loaded using pool lzo/zbud [ 3.570288] Key type big_key registered [ 3.581564] Key type encrypted registered [ 3.583173] ima: No TPM chip found, activating TPM-bypass! [ 3.584684] ima: Allocated hash algorithm: sha1 [ 3.585987] ima: No architecture policies found [ 3.587355] evm: Initialising EVM extended attributes: [ 3.588848] evm: security.selinux [ 3.589915] evm: security.ima [ 3.590811] evm: security.capability [ 3.591756] evm: HMAC attrs: 0x1 [ 3.594203] rtc_cmos 00:05: setting system clock to 2026-09-03 12:44:37 UTC (1788439477) [ 3.600336] debug: unmapping init [mem 0xffffffffb1803000-0xffffffffb19fffff] [ 3.602969] debug: unmapping init [mem 0xffffffffb0582000-0xffffffffb0858fff] [ 3.607061] Write protecting the kernel read-only data: 28672k [ 3.610424] debug: unmapping init [mem 0xffffffffaec03000-0xffffffffaedfffff] [ 3.613023] debug: unmapping init [mem 0xffffffffaf514000-0xffffffffaf5fffff] [ 3.646505] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.656088] systemd[1]: Detected virtualization kvm. [ 3.657527] systemd[1]: Detected architecture x86-64. [ 3.658720] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.688527] systemd[1]: No hostname configured. [ 3.691047] systemd[1]: Set hostname to . [ 3.693809] random: systemd: uninitialized urandom read (16 bytes read) [ 3.696570] systemd[1]: Initializing machine ID from random generator. [ 3.745756] random: ln: uninitialized urandom read (6 bytes read) [ 3.834657] random: systemd: uninitialized urandom read (16 bytes read) [ 3.837611] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.845227] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.850990] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. [ OK ] Reached target Sockets. Starting Journal Service... Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.461939] device-mapper: uevent: version 1.0.3 [ 4.465307] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK [ 5.276021] random: fast init done ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.318964] virtio_net virtio0 ens2: renamed from eth0 [ 5.447344] scsi host0: ata_piix [ 5.464314] scsi host1: ata_piix [ 5.466407] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.470443] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.252742] dracut-initqueue[586]: RTNETLINK answers: File exists [ 10.098231] random: crng init done [ 10.099905] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.686768] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.948539] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.259471] SELinux: Disabled at runtime. [ 12.325717] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.334837] systemd[1]: Detected virtualization kvm. [ 12.336858] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.970814] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.977267] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.985491] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.990438] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.994325] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.003376] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.009369] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Reached target rpc_pipefs.target. Starting Remount Root and Kernel File Systems... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ 13.168694] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [ OK ] Stopped target Initrd File Systems. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 13.577138] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.970929] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.972717] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.126095] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.139498] EDAC sbridge: Ver: 1.1.2 [ 16.233656] Key type dns_resolver registered [ 16.547130] NFS: Registering the id_resolver key type [ 16.549079] Key type id_resolver registered [ 16.550708] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started RPC Bind. [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Network Manager... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg457-server login: [ 40.838010] hrtimer: interrupt took 26001150 ns [ 82.975434] libcfs: loading out-of-tree module taints kernel. [ 83.001111] Key type ._llcrypt registered [ 83.003956] Key type .llcrypt registered [ 83.142359] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [ 105.115539] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 107.461777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 107.515137] alg: No test for adler32 (adler32-zlib) [ 110.023910] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 112.518848] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 114.591661] Key type lgssc registered [ 116.854197] Lustre: Echo OBD driver; http://www.lustre.org/ [ 137.969622] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 184.080822] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 199.873503] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 199.902550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 201.333331] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 201.385988] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 201.485291] Lustre: lustre-MDT0000: new disk, initializing [ 201.633182] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 201.661381] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 207.092597] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 222.865235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 222.991296] Lustre: 6517:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 223.060550] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 223.071400] Lustre: Skipped 1 previous similar message [ 223.283802] Lustre: lustre-MDT0001: new disk, initializing [ 223.432460] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 223.485652] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 223.511846] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 228.825983] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 234.434883] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 246.526536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 246.804426] Lustre: lustre-OST0000: new disk, initializing [ 246.807310] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 246.813627] Lustre: 8453:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 246.876069] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 248.427126] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 248.442154] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 248.553266] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 254.520622] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 271.470213] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 271.690949] Lustre: lustre-OST0001: new disk, initializing [ 271.695100] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 271.706710] Lustre: 9528:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 271.819462] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 277.565737] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 277.576366] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 277.724722] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 279.189660] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 292.641106] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 300.460084] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 307.719832] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing check_logdir /tmp/testlogs/ [ 313.047747] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing yml_node [ 318.880700] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 322.082498] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 325.290811] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 327.347649] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 3 08:49:58 EDT 2026 [ 345.954199] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 347.704559] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 365.024328] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 365.037229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 365.054358] Lustre: Skipped 3 previous similar messages [ 365.079218] Lustre: Skipped 3 previous similar messages [ 369.872651] Lustre: server umount lustre-MDT0000 complete [ 374.222304] LustreError: 7464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439848 with bad export cookie 5286637577018625421 [ 374.242866] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 375.268241] LustreError: 6523:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 375.271793] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 375.295847] LustreError: 6523:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 375.300893] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 379.295982] LustreError: 10353:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 379.296975] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 379.323359] LustreError: 10353:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 379.352131] Lustre: Skipped 2 previous similar messages [ 380.748495] Lustre: server umount lustre-MDT0001 complete [ 385.401226] Lustre: server umount lustre-OST0000 complete [ 389.468316] Lustre: server umount lustre-OST0001 complete [ 397.735828] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [ 410.172819] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 452.686511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 453.159346] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 453.199079] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 453.329322] Lustre: lustre-MDT0000: new disk, initializing [ 453.485766] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 453.508179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 459.763043] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 471.201753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 471.369299] Lustre: 15081:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 471.450242] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 471.452564] Lustre: Skipped 1 previous similar message [ 471.556812] Lustre: lustre-MDT0001: new disk, initializing [ 471.732321] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 471.804235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 471.821880] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 477.400202] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 489.739650] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 498.101243] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 505.391281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 505.762128] Lustre: lustre-OST0000: new disk, initializing [ 505.767714] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 505.783570] Lustre: 17049:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 505.906772] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 506.908392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 506.919947] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 507.073069] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 513.219783] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 521.804647] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 526.564791] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 526.743781] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 531.225890] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 531.523823] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 537.579904] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 537.605177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 537.609572] Lustre: Skipped 2 previous similar messages [ 537.633178] Lustre: Skipped 1 previous similar message [ 542.704171] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 542.707751] Lustre: Skipped 1 previous similar message [ 548.319153] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 548.516341] Lustre: server umount lustre-OST0000 complete [ 553.955600] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 553.974969] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 553.989338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 554.005696] Lustre: Skipped 2 previous similar messages [ 558.055086] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 559.371455] Lustre: server umount lustre-MDT0000 complete [ 563.168271] LustreError: 15090:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 563.187488] LustreError: 15090:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 563.281382] LustreError: 15072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440037 with bad export cookie 5286637577018626954 [ 563.294707] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 563.295224] LustreError: 15072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 563.975174] Lustre: server umount lustre-MDT0001 complete [ 570.281899] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 08:54:01 (1788440041) [ 589.470355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 589.963671] LustreError: 19925:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 590.041620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 595.060169] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 595.426227] LustreError: 19926:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 595.446055] LustreError: 19926:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 604.648207] LustreError: 19926:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 604.667934] LustreError: 19926:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 605.396781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 610.602677] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 620.087519] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 626.938926] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 628.891843] Lustre: Failing over lustre-MDT0000 [ 629.141889] Lustre: server umount lustre-MDT0000 complete [ 630.956300] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 630.958329] LustreError: 19925:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 630.983462] Lustre: Skipped 1 previous similar message [ 646.627020] Lustre: 3648:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788440104/real 1788440104] req@ffff9819bc2dc000 x1875314827499392/t0(0) o400->MGC192.168.204.157@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788440120 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 646.652809] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 650.260989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 657.975558] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x495de8ddd26d5910 [ 657.992585] Lustre: MGC192.168.204.157@tcp: Connection restored to 0@lo (at 0@lo) [ 658.313934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 658.325323] Lustre: Skipped 1 previous similar message [ 658.420029] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 663.017854] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 663.527658] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 663.529058] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 663.566048] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 674.023912] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 678.883891] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 678.892156] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 678.907866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 678.910126] Lustre: Skipped 2 previous similar messages [ 684.266352] Lustre: server umount lustre-MDT0000 complete [ 686.055115] LustreError: 20653:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 686.091732] LustreError: 20653:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 16 previous similar messages [ 688.453042] LustreError: 20660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440162 with bad export cookie 5286637577018628368 [ 688.459294] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 688.463408] LustreError: 20660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 688.846324] Lustre: server umount lustre-MDT0001 complete [ 698.292035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 704.423586] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 713.558559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 719.319267] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 727.761826] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 734.425098] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 736.864353] Lustre: Failing over lustre-MDT0000 [ 737.250224] Lustre: server umount lustre-MDT0000 complete [ 738.788305] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 738.805713] Lustre: Skipped 2 previous similar messages [ 750.049098] LustreError: 23267:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 750.069456] LustreError: 23267:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 754.156700] Lustre: 3647:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788440212/real 1788440212] req@ffff9819bc2c0e00 x1875314827529344/t0(0) o400->MGC192.168.204.157@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788440228 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 754.199466] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 757.255496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 764.414532] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x495de8ddd26d5cba [ 764.432777] Lustre: MGC192.168.204.157@tcp: Connection restored to 0@lo (at 0@lo) [ 764.440728] Lustre: Skipped 1 previous similar message [ 764.811960] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 770.030341] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 770.041153] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 770.066485] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 770.394605] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 781.360373] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 790.510611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 790.518913] Lustre: Skipped 2 previous similar messages [ 790.519162] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 792.263732] Lustre: server umount lustre-MDT0000 complete [ 796.785936] LustreError: 23247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440270 with bad export cookie 5286637577018629306 [ 796.800355] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 796.805389] LustreError: 23247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 797.273115] Lustre: server umount lustre-MDT0001 complete [ 807.561045] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 08:57:58 (1788440278) [ 825.710598] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [ 835.276223] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 878.257934] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 887.246686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 887.494695] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 887.530621] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 887.633654] Lustre: lustre-MDT0000: new disk, initializing [ 887.743285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 887.751046] Lustre: Skipped 3 previous similar messages [ 887.784547] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 892.187210] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 904.753204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 905.161126] Lustre: 29642:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 905.669562] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 911.144780] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 920.942237] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 926.671871] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 934.041669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 934.319781] Lustre: 31612:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 935.841170] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 935.911598] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 936.002253] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 936.029600] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 941.225306] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 950.657546] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 956.646739] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 976.867210] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 976.869483] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 976.876995] Lustre: Skipped 2 previous similar messages [ 976.887697] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 976.887706] Lustre: Skipped 1 previous similar message [ 991.199243] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 991.502388] Lustre: server umount lustre-OST0000 complete [ 1002.170143] Lustre: server umount lustre-MDT0000 complete [ 1002.479863] LustreError: 30496:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1002.510441] LustreError: 30496:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 1006.588268] LustreError: 30438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440480 with bad export cookie 5286637577018629719 [ 1006.600212] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1006.605099] LustreError: 30438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1007.209654] Lustre: server umount lustre-MDT0001 complete [ 1019.019486] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 1022.349558] Key type lgssc unregistered [ 1022.756841] LNet: 33607:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1022.774908] LNetError: 33607:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1022.800550] LNet: Removed LNI 192.168.204.157@tcp [ 1023.982175] Key type .llcrypt unregistered [ 1023.983954] Key type ._llcrypt unregistered [ 1052.835912] Key type ._llcrypt registered [ 1052.839640] Key type .llcrypt registered [ 1052.940541] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 1054.444422] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1054.548310] alg: No test for adler32 (adler32-zlib) [ 1055.698218] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1055.973876] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 1057.655210] Key type lgssc registered [ 1059.133398] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1067.875585] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1067.890584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1069.381577] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1074.406141] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1079.784340] LustreError: 35020:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1079.806118] LustreError: 35020:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1084.737856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1084.905321] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1085.062832] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1089.831816] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1098.496942] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1105.213844] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1111.353916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1111.868797] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1114.974663] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1118.908095] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1128.175369] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1129.263107] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1144.225899] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 09:03:35 (1788440615) [ 1146.065455] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1148.041152] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1154.569525] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 09:03:45 (1788440625) [ 1165.766403] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 09:03:56 (1788440636) [ 1166.713989] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1176.630618] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1176.638752] Lustre: Skipped 2 previous similar messages [ 1178.561791] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 09:04:09 (1788440649) [ 1180.712860] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1180.719749] Lustre: Skipped 2 previous similar messages [ 1190.576218] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1190.585741] Lustre: Skipped 4 previous similar messages [ 1206.795725] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 09:04:38 (1788440678) [ 1209.613280] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1213.365460] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1222.085305] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 09:04:53 (1788440693) [ 1223.995463] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1224.004446] Lustre: Skipped 6 previous similar messages [ 1237.328812] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 09:05:08 (1788440708) [ 1253.349111] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1253.372074] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1253.416764] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1257.460996] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1257.465035] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1257.497773] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1259.484943] Lustre: server umount lustre-OST0000 complete [ 1264.099544] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1264.119956] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1264.157288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1264.173136] Lustre: Skipped 1 previous similar message [ 1267.681862] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1270.115956] Lustre: server umount lustre-MDT0000 complete [ 1272.800308] LustreError: 35020:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1272.830567] LustreError: 35020:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1273.652958] LustreError: 37939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440747 with bad export cookie 7160379281359653418 [ 1273.657300] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1273.672544] LustreError: 37939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1274.030045] Lustre: server umount lustre-MDT0001 complete [ 1287.592322] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 1291.211978] Key type lgssc unregistered [ 1291.682944] LNet: 42884:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1291.697700] LNetError: 42884:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1291.709858] LNet: Removed LNI 192.168.204.157@tcp [ 1293.065110] Key type .llcrypt unregistered [ 1293.068320] Key type ._llcrypt unregistered [ 1313.587928] Key type ._llcrypt registered [ 1313.590983] Key type .llcrypt registered [ 1313.716313] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [ 1331.593708] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 1333.191095] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1333.211599] alg: No test for adler32 (adler32-zlib) [ 1334.292195] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1334.559886] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 1336.303401] Key type lgssc registered [ 1337.573426] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1374.238437] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1374.260705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1375.663315] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1375.721984] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1375.873578] Lustre: lustre-MDT0000: new disk, initializing [ 1376.039419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1376.080409] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1381.799336] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1393.187630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1393.293406] Lustre: 46037:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1393.372552] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1393.378673] Lustre: Skipped 1 previous similar message [ 1393.461957] Lustre: lustre-MDT0001: new disk, initializing [ 1393.603863] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1393.647500] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1393.665684] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1398.621800] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1408.641926] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1415.071489] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1422.246880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1422.549633] Lustre: lustre-OST0000: new disk, initializing [ 1422.556479] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1422.569767] Lustre: 48005:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1422.685982] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1429.482227] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1432.148823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1432.161913] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1432.260627] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1439.442807] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1444.693757] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1444.963462] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1449.798770] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1450.005450] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1453.033045] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1453.034548] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1453.054685] LustreError: Skipped 1 previous similar message [ 1453.061359] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1453.085651] Lustre: Skipped 1 previous similar message [ 1453.125983] Lustre: Skipped 1 previous similar message [ 1458.148816] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1458.158191] Lustre: Skipped 1 previous similar message [ 1463.281531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1463.292069] Lustre: Skipped 1 previous similar message [ 1466.335155] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1466.519383] Lustre: server umount lustre-OST0000 complete [ 1470.444070] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1470.453995] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1470.467131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1473.520101] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1476.750342] Lustre: server umount lustre-MDT0000 complete [ 1480.171273] LustreError: 46030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440954 with bad export cookie 11810297899553492985 [ 1480.177457] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1480.182875] LustreError: 46030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1480.457445] Lustre: server umount lustre-MDT0001 complete [ 1491.044966] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 09:09:21 (1788440961) [ 1498.160660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1498.755890] LustreError: 50466:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1498.777478] LustreError: 50466:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1498.895325] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1504.148661] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1504.226310] LustreError: 50465:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1508.332319] LustreError: 50466:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1513.437214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1513.442259] LustreError: 50465:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1514.004760] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1519.644714] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1527.447205] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1533.912958] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1541.882641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1542.220720] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1548.564972] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1555.614622] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1564.352708] LustreError: 52994:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1565.084942] Lustre: Modifying parameter general.timeout=1 in log params [ 1566.838725] Lustre: Modifying parameter general.timeout=3 in log params [ 1566.843742] Lustre: Skipped 1 previous similar message [ 1569.416906] Lustre: Modifying parameter general.timeout=6 in log params [ 1569.424276] Lustre: Skipped 2 previous similar messages [ 1573.903931] Lustre: Modifying parameter general.timeout=10 in log params [ 1573.907335] Lustre: Skipped 3 previous similar messages [ 1582.456501] Lustre: Modifying parameter general.timeout=20 in log params [ 1582.463474] Lustre: Skipped 9 previous similar messages [ 1598.460755] Lustre: Modifying parameter general.timeout=38 in log params [ 1598.465763] Lustre: Skipped 17 previous similar messages [ 1631.123401] Lustre: Modifying parameter general.timeout=72 in log params [ 1631.128487] Lustre: Skipped 33 previous similar messages [ 1696.901947] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 09:12:48 (1788441168) [ 1698.160942] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_policies=tbf in log params [ 1698.167834] Lustre: Skipped 58 previous similar messages [ 1699.989949] Lustre: 59947:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1701.859232] Lustre: 60046:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1703.892575] Lustre: 60144:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1709.497971] Lustre: 60464:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1709.508725] Lustre: 60464:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1736.075676] Lustre: 61823:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1736.112421] LustreError: 61823:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1740.023725] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 09:13:31 (1788441211) [ 1759.728785] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1759.731243] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1759.742552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1759.742562] Lustre: Skipped 2 previous similar messages [ 1759.774510] Lustre: Skipped 2 previous similar messages [ 1761.162683] Lustre: server umount lustre-MDT0000 complete [ 1764.842676] LustreError: 50465:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1764.868690] LustreError: 50465:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1764.964221] LustreError: 50445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441238 with bad export cookie 11810297899553493867 [ 1764.974767] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1764.975606] LustreError: 50445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1765.339647] Lustre: server umount lustre-MDT0001 complete [ 1779.180433] Lustre: server umount lustre-OST0000 complete [ 1811.811110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1812.346362] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1812.760585] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1817.059716] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1826.913240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1827.136564] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1827.219262] Lustre: 64359:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1833.886989] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1843.013386] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1849.858433] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1857.736825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1857.986097] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1858.264439] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1858.271892] Lustre: Skipped 1 previous similar message [ 1865.410936] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1874.411654] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1876.492115] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1876.507188] Lustre: Skipped 22 previous similar messages [ 1876.576077] Lustre: 66923:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1886.690258] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1886.698519] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1886.705775] LustreError: Skipped 1 previous similar message [ 1886.711848] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1886.737514] Lustre: Skipped 1 previous similar message [ 1886.745189] Lustre: Skipped 2 previous similar messages [ 1891.263938] Lustre: server umount lustre-OST0000 complete [ 1896.941413] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1906.149412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1906.162221] Lustre: Skipped 6 previous similar messages [ 1910.241318] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1910.573583] Lustre: server umount lustre-MDT0000 complete [ 1911.264979] LustreError: 64364:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1911.289788] LustreError: 64364:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1914.882924] LustreError: 64350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441388 with bad export cookie 11810297899553496947 [ 1914.891612] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1914.901573] LustreError: 64350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1915.362682] Lustre: server umount lustre-MDT0001 complete [ 1928.800864] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 1931.908371] Key type lgssc unregistered [ 1932.265336] LNet: 68131:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1932.273616] LNetError: 68131:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1932.308930] LNet: Removed LNI 192.168.204.157@tcp [ 1933.334255] Key type .llcrypt unregistered [ 1933.338250] Key type ._llcrypt unregistered [ 1960.622994] Key type ._llcrypt registered [ 1960.629937] Key type .llcrypt registered [ 1960.789959] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 1962.208021] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1962.310383] alg: No test for adler32 (adler32-zlib) [ 1963.698795] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1964.106511] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 1965.848410] Key type lgssc registered [ 1967.372237] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1975.671867] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1975.696275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1977.434179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1983.265448] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1987.559304] LustreError: 69543:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1987.574799] LustreError: 69543:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1992.682313] LustreError: 69544:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1993.084412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1993.669989] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1998.657265] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2007.074968] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2013.977923] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2022.027892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2022.322279] LustreError: 71372:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2022.722915] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2029.833092] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2038.546809] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2047.615298] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2058.724880] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2058.731135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2058.739785] Lustre: Skipped 2 previous similar messages [ 2058.763704] Lustre: Skipped 2 previous similar messages [ 2063.841816] LustreError: 70276:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2063.872694] LustreError: 70276:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2063.985395] Lustre: server umount lustre-MDT0000 complete [ 2067.594453] LustreError: 72228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441541 with bad export cookie 4306008778441365714 [ 2067.598500] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2067.608064] LustreError: 72228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2067.931586] Lustre: server umount lustre-MDT0001 complete [ 2081.472770] Lustre: server umount lustre-OST0000 complete [ 2111.869902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2112.288618] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2112.338450] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2112.613568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2118.017456] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2127.797409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2127.954954] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2127.987189] Lustre: 74417:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2128.505879] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2133.223871] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2141.132293] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2147.316659] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2153.998498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2154.188077] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2154.554297] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2160.559767] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2168.654292] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2170.540789] Lustre: 76994:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2172.305184] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2179.562271] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2179.577394] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2179.596486] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2180.065944] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2183.851277] Lustre: server umount lustre-OST0000 complete [ 2189.799840] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2189.809964] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2189.826372] Lustre: Skipped 1 previous similar message [ 2189.833740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2189.838314] Lustre: Skipped 2 previous similar messages [ 2194.215619] Lustre: server umount lustre-MDT0000 complete [ 2195.424209] LustreError: 75167:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2195.443230] LustreError: 75167:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2198.243386] LustreError: 74408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441672 with bad export cookie 4306008778441366561 [ 2198.256098] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2198.269779] LustreError: 74408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2198.850489] Lustre: server umount lustre-MDT0001 complete [ 2211.571085] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 2214.488261] Key type lgssc unregistered [ 2214.770562] LNet: 78205:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2214.781922] LNetError: 78205:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2214.803401] LNet: Removed LNI 192.168.204.157@tcp [ 2215.614561] Key type .llcrypt unregistered [ 2215.617583] Key type ._llcrypt unregistered [ 2240.116885] Key type ._llcrypt registered [ 2240.121048] Key type .llcrypt registered [ 2240.294417] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 2241.895422] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2241.942730] alg: No test for adler32 (adler32-zlib) [ 2243.227108] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2243.448443] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 2245.186357] Key type lgssc registered [ 2246.755335] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2255.099132] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2255.124861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2256.798688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2262.330284] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2267.122529] LustreError: 79613:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2267.155701] LustreError: 79613:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2271.689606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2272.339165] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2276.409169] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2284.060911] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2290.870771] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2299.437367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2300.190033] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2308.091966] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2315.458618] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2320.978899] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2321.806365] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2323.642182] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2323.648966] Lustre: Skipped 1 previous similar message [ 2326.269868] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2326.275948] Lustre: Skipped 2 previous similar messages [ 2330.303852] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2330.310238] Lustre: Skipped 4 previous similar messages [ 2338.746041] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2338.751434] Lustre: Skipped 9 previous similar messages [ 2355.278778] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2355.285092] Lustre: Skipped 16 previous similar messages [ 2387.348740] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2387.352718] Lustre: Skipped 35 previous similar messages [ 2419.489509] Lustre: 87718:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2422.333749] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 09:24:53 (1788441893) [ 2423.972335] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2426.059381] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 09:24:57 (1788441897) [ 2434.529222] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2434.530599] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2434.546355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2434.565463] Lustre: Skipped 1 previous similar message [ 2436.748050] Lustre: server umount lustre-OST0000 complete [ 2441.187828] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2441.201101] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2441.222556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2441.236672] Lustre: Skipped 1 previous similar message [ 2444.775495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2444.777327] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2447.385728] Lustre: server umount lustre-MDT0000 complete [ 2449.889888] LustreError: 79613:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2449.907611] LustreError: 79613:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2451.381709] LustreError: 82747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441925 with bad export cookie 10819875274838928450 [ 2451.386053] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2451.389939] LustreError: 82747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2451.953264] Lustre: server umount lustre-MDT0001 complete [ 2464.353071] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 2466.719176] Key type lgssc unregistered [ 2466.921806] LNet: 88966:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2466.934884] LNetError: 88966:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2466.949108] LNet: Removed LNI 192.168.204.157@tcp [ 2467.683965] Key type .llcrypt unregistered [ 2467.686229] Key type ._llcrypt unregistered [ 2471.503555] Key type ._llcrypt registered [ 2471.508721] Key type .llcrypt registered [ 2471.613353] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2476.545894] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules [ 2478.132315] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2478.154959] alg: No test for adler32 (adler32-zlib) [ 2479.290209] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2479.780423] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 2481.621726] Key type lgssc registered [ 2483.288683] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2484.703652] LustreError: 90205:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2504.021879] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 2507.500817] LustreError: 90205:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2514.082713] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2514.113386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2515.740768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2520.939978] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2525.154478] LustreError: 91239:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2525.186601] LustreError: 91239:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2530.080908] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 09:26:41 (1788442001) [ 2530.279441] LustreError: 91240:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2534.662289] LustreError: 91271:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2534.675673] LustreError: 91271:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2534.683733] LustreError: 91271:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 2534.949974] Lustre: server umount lustre-MDT0000 complete [ 2549.434759] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 2552.307028] Key type lgssc unregistered [ 2552.655728] LNet: 92766:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2552.676883] LNetError: 92766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2552.702108] LNet: Removed LNI 192.168.204.157@tcp [ 2553.608663] Key type .llcrypt unregistered [ 2553.612728] Key type ._llcrypt unregistered [ 2570.275645] Key type ._llcrypt registered [ 2570.279717] Key type .llcrypt registered [ 2570.417536] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 2571.147720] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2571.166189] alg: No test for adler32 (adler32-zlib) [ 2572.388772] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2572.947302] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 2574.660744] Key type lgssc registered [ 2576.167674] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2586.069442] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2586.094438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2587.723555] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2593.155483] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2597.861290] LustreError: 93980:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2597.880229] LustreError: 93980:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2600.662219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2601.897294] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2605.772834] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2613.093695] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2618.769859] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2626.104536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2626.924215] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2634.122757] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2642.496714] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2693.903342] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 09:29:25 (1788442165) [ 2696.036823] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2698.050381] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 09:29:29 (1788442169) [ 2703.841056] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2703.851083] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2703.863272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2704.873625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2704.873644] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2704.906758] Lustre: Skipped 1 previous similar message [ 2708.511190] Lustre: server umount lustre-MDT0000 complete [ 2709.987471] LustreError: 93981:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2710.014037] LustreError: 93981:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2712.710040] LustreError: 93960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442186 with bad export cookie 10329363794096226686 [ 2712.711540] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2712.725821] LustreError: 93960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2713.088426] Lustre: server umount lustre-MDT0001 complete [ 2727.936156] Lustre: server umount lustre-OST0000 complete [ 2742.183683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2742.877824] LustreError: 98281:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2743.051339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2748.386713] LustreError: 98282:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2748.415439] LustreError: 98282:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2749.126637] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2752.509360] LustreError: 98281:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2758.785410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2759.639573] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2765.191988] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2773.541382] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2779.986699] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2792.847654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2793.105298] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2793.115440] LustreError: 98268:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2793.150142] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2793.160969] LustreError: 100366:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -98 [ 2793.166955] LustreError: 100366:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2793.172477] LustreError: 100366:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2793.274718] Lustre: server umount lustre-OST0000 complete [ 2793.277958] LustreError: 100366:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2802.282753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2802.449598] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2802.455844] LustreError: 98271:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2802.474403] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2802.481397] LustreError: 100833:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -98 [ 2802.491939] LustreError: 100833:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2802.497254] LustreError: 100833:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2802.573824] Lustre: server umount lustre-OST0000 complete [ 2802.578496] LustreError: 100833:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2813.007476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2813.250191] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2813.254142] Lustre: Found index 0 for lustre-OST0000, updating log [ 2813.260976] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2813.352824] Lustre: lustre-OST0000: new disk, initializing [ 2813.359753] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2813.369465] Lustre: 101368:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2813.729192] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2821.539433] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2825.839636] Lustre: Failing over lustre-OST0000 [ 2826.063750] Lustre: server umount lustre-OST0000 complete [ 2830.097078] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2830.110028] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2830.112810] Lustre: Skipped 1 previous similar message [ 2830.136610] Lustre: Skipped 3 previous similar messages [ 2835.425375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2835.436984] Lustre: Skipped 1 previous similar message [ 2836.495350] Lustre: server umount lustre-MDT0000 complete [ 2840.544187] LustreError: 99013:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2840.565259] LustreError: 99013:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2840.811559] LustreError: 99023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442314 with bad export cookie 10329363794096229430 [ 2840.818238] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2840.826844] LustreError: 99023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2841.278763] Lustre: server umount lustre-MDT0001 complete [ 2850.276631] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 09:32:01 (1788442321) [ 2868.937210] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 2879.134760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2879.872774] LustreError: 103714:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2880.107164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2885.933700] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2895.496155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2900.473400] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2902.962610] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2909.299533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2916.309178] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2924.608667] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2924.708762] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2924.775406] Lustre: lustre-OST0001: new disk, initializing [ 2924.778318] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2924.786528] Lustre: 106033:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2925.081200] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2925.089384] Lustre: Skipped 2 previous similar messages [ 2931.967757] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2933.906566] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2933.919547] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2934.054333] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2940.770679] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2948.881298] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2954.723221] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2954.734075] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2954.747168] Lustre: Skipped 2 previous similar messages [ 2954.762192] Lustre: Skipped 1 previous similar message [ 2959.848378] LustreError: 103714:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2959.863612] LustreError: 103714:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 2960.013439] Lustre: server umount lustre-MDT0001 complete [ 2969.687811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2969.921707] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2969.938264] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2970.016357] Lustre: 103700:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2971.159069] LustreError: 107570:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2971.174122] LustreError: 107570:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2971.180333] LustreError: 103705:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2971.271874] LustreError: 107578:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.157@tcp: cfg command failed: rc = -17 [ 2971.281255] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2971.281255] [ 2975.543429] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2976.245168] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2985.042756] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 09:34:16 (1788442456) [ 3003.164437] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 3022.454949] Lustre: 109649:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3045.445781] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3048.941338] Lustre: 110784:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3134.433048] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3134.445334] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3134.461738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3134.465894] Lustre: Skipped 2 previous similar messages [ 3138.671979] Lustre: server umount lustre-MDT0000 complete [ 3140.067959] LustreError: 104973:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3140.087668] LustreError: 104973:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 3142.840817] LustreError: 103694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442616 with bad export cookie 10329363794096230424 [ 3142.843925] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3142.858824] LustreError: 103694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3143.283440] Lustre: server umount lustre-MDT0001 complete [ 3157.659630] Lustre: server umount lustre-OST0000 complete [ 3172.017526] Lustre: server umount lustre-OST0001 complete [ 3178.140550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3188.257057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3197.346559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3206.587783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3226.438682] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 3238.315362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3238.337323] Lustre: lustre-MDT0000: reset Object Index mappings [ 3239.205552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3239.208115] Lustre: Skipped 1 previous similar message [ 3244.479655] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3254.941111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3254.974406] Lustre: lustre-MDT0001: reset Object Index mappings [ 3260.648045] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3263.918879] Lustre: 115880:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3271.041829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3278.080464] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3281.023180] LustreError: 116235:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3281.042362] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 3281.048559] LustreError: 116235:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 3283.148429] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 3288.021873] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3294.204483] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3294.210481] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3296.245622] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3305.069711] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3309.505541] Lustre: 117759:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3362.522869] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 09:40:33 (1788442833) [ 3388.897744] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3388.915714] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3388.941112] Lustre: Skipped 3 previous similar messages [ 3388.949300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3388.959623] Lustre: Skipped 3 previous similar messages [ 3394.087808] Lustre: server umount lustre-MDT0000 complete [ 3398.179460] LustreError: 117761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442872 with bad export cookie 10329363794096277254 [ 3398.191839] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3398.193903] LustreError: 117761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3404.854250] Lustre: server umount lustre-MDT0001 complete [ 3409.166400] LustreError: 118520:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0000_UUID: namespace resource [0x280000400:0x7f:0x0].0x0 (ffff9819a6c5d200) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 3415.366969] Lustre: server umount lustre-OST0000 complete [ 3425.364968] Lustre: server umount lustre-OST0001 complete [ 3433.736405] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [ 3443.805643] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 3488.988764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3489.263857] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3489.288899] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3489.384416] Lustre: lustre-MDT0000: new disk, initializing [ 3489.462590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3489.467414] Lustre: Skipped 3 previous similar messages [ 3489.480580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3494.508018] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3501.873183] Lustre: server umount lustre-MDT0000 complete [ 3512.647219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3513.101362] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3513.136142] Lustre: 122030:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3513.159726] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3518.324955] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3523.629312] Lustre: server umount lustre-MDT0000 complete [ 3533.375589] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 09:43:24 (1788443004) [ 3535.350715] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3536.983202] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 09:43:28 (1788443008) [ 3555.993437] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 3566.206028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3571.294332] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3581.115952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3581.246775] Lustre: 124081:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3581.333804] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3581.338697] Lustre: Skipped 1 previous similar message [ 3581.454612] Lustre: lustre-MDT0001: new disk, initializing [ 3581.676490] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3581.700558] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3586.812892] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3592.029800] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3599.801075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3600.179898] Lustre: lustre-OST0000: new disk, initializing [ 3600.183595] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3600.189608] Lustre: 125662:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3602.204347] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3602.214294] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3602.421698] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3607.680559] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3620.107800] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3620.370996] Lustre: lustre-OST0001: new disk, initializing [ 3620.373754] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3620.377818] Lustre: 126533:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3622.326444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3622.341804] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3622.485185] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3627.851593] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3638.601850] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3642.303845] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4041.270726] Lustre: 132952:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4041.283774] Lustre: 132952:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4041.289511] Lustre: 132952:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4041.297388] Lustre: 132952:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4041.313377] Lustre: 132952:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4041.319810] Lustre: 132952:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4324.735439] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 09:56:34 (1788443794) [ 4342.454696] Lustre: server umount lustre-MDT0000 complete [ 4344.302489] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4344.308624] LustreError: 124091:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4344.312237] Lustre: Skipped 8 previous similar messages [ 4344.351512] LustreError: 124091:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 4346.676565] LustreError: 132964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443820 with bad export cookie 10329363794096345161 [ 4346.700583] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4349.409131] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4349.414338] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4349.429246] Lustre: Skipped 1 previous similar message [ 4349.443347] Lustre: Skipped 9 previous similar messages [ 4353.315875] Lustre: server umount lustre-MDT0001 complete [ 4364.186355] Lustre: server umount lustre-OST0000 complete [ 4374.750365] Lustre: server umount lustre-OST0001 complete [ 4381.841515] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [ 4393.490452] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 4432.044709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4432.334304] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4432.399287] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4432.522544] Lustre: lustre-MDT0000: new disk, initializing [ 4432.613346] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4432.621134] Lustre: Skipped 5 previous similar messages [ 4432.652753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4438.521913] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 4449.904528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4450.033871] Lustre: 136002:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4450.123732] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4450.128049] Lustre: Skipped 1 previous similar message [ 4450.283427] Lustre: lustre-MDT0001: new disk, initializing [ 4450.553155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4450.573629] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4455.743209] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 4465.839842] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4472.122368] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4479.012559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4479.278339] Lustre: lustre-OST0000: new disk, initializing [ 4479.284038] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4479.298776] Lustre: 137972:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4479.465104] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4479.473511] Lustre: Skipped 1 previous similar message [ 4481.236236] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4481.252194] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4481.365613] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4487.024171] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 4496.824117] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4507.518316] Lustre: lustre-MDD0000: changelog on [ 4510.491283] Lustre: lustre-MDD0001: changelog on [ 6408.594963] Lustre: lustre-MDD0001: changelog off [ 6410.373534] Lustre: 139428:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6411.161195] Lustre: lustre-MDD0000: changelog off [ 6417.378831] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6417.389614] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6417.399634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6417.402312] Lustre: Skipped 2 previous similar messages [ 6419.939919] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6419.940724] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6419.983080] Lustre: Skipped 1 previous similar message [ 6425.078782] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6425.090423] Lustre: Skipped 1 previous similar message [ 6430.181958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6430.186010] Lustre: Skipped 1 previous similar message [ 6431.199170] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6431.396813] Lustre: server umount lustre-OST0000 complete [ 6435.303425] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6435.322327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6435.324827] Lustre: Skipped 1 previous similar message [ 6435.337480] Lustre: Skipped 1 previous similar message [ 6441.041103] Lustre: server umount lustre-MDT0000 complete [ 6444.848263] LustreError: 135994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445918 with bad export cookie 10329363794097164560 [ 6444.854911] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6444.866616] LustreError: 135994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6445.532269] Lustre: server umount lustre-MDT0001 complete [ 6455.271362] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 6459.120262] Key type lgssc unregistered [ 6459.568245] LNet: 147122:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6459.576609] LNetError: 147122:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6459.596832] LNet: Removed LNI 192.168.204.157@tcp [ 6460.621134] Key type .llcrypt unregistered [ 6460.626642] Key type ._llcrypt unregistered [ 6484.327071] Key type ._llcrypt registered [ 6484.328767] Key type .llcrypt registered [ 6484.439658] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [ 6500.251586] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 6500.986690] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6501.021820] alg: No test for adler32 (adler32-zlib) [ 6502.108940] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 6502.326429] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 6504.039226] Key type lgssc registered [ 6505.159259] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6540.672075] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6540.714377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6542.020527] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6542.068377] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6542.207239] Lustre: lustre-MDT0000: new disk, initializing [ 6542.293540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6542.312741] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6547.041425] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 6557.374279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6557.470285] Lustre: 150335:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6557.514067] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6557.518276] Lustre: Skipped 1 previous similar message [ 6557.580452] Lustre: lustre-MDT0001: new disk, initializing [ 6557.671489] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6557.719667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6557.742074] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6562.427472] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 6571.580171] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6577.416159] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6588.307973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6588.644506] Lustre: lustre-OST0000: new disk, initializing [ 6588.650703] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6588.662677] Lustre: 152306:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6588.808070] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6597.220498] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6597.251804] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6597.454753] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6598.467216] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 6607.776673] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6614.447063] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_module obdecho/obdecho [ 6616.375822] LustreError: 153310:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6616.385732] LustreError: 153310:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6618.370552] LustreError: 153406:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6629.350557] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6629.364444] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6629.382636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6633.446918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6633.450457] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6633.458755] Lustre: Skipped 1 previous similar message [ 6633.475148] Lustre: Skipped 1 previous similar message [ 6635.609549] Lustre: server umount lustre-MDT0000 complete [ 6639.524566] LustreError: 152305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446113 with bad export cookie 806024051905664668 [ 6639.531861] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6639.540968] LustreError: 152305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6639.966408] Lustre: server umount lustre-MDT0001 complete [ 6655.139831] Lustre: server umount lustre-OST0000 complete [ 6678.307476] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [ 6688.526855] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 6726.207417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6726.473303] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6726.505512] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6726.601991] Lustre: lustre-MDT0000: new disk, initializing [ 6726.768170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6726.788124] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6732.009341] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 6742.511755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6742.668058] Lustre: 157122:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6742.751541] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6742.755742] Lustre: Skipped 1 previous similar message [ 6742.844801] Lustre: lustre-MDT0001: new disk, initializing [ 6743.011811] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6743.075347] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6743.088791] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6747.683334] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 6758.347856] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6764.859370] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6772.368528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6772.642083] Lustre: lustre-OST0000: new disk, initializing [ 6772.650660] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6772.662075] Lustre: 159091:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6772.748857] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6779.262557] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 6780.977393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6780.988691] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6781.126854] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6789.537089] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6794.571047] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6794.825665] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6800.152368] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6800.457552] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6807.011371] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6807.011990] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6807.020508] LustreError: Skipped 1 previous similar message [ 6807.021977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6807.021984] Lustre: Skipped 1 previous similar message [ 6807.053974] Lustre: Skipped 1 previous similar message [ 6808.832392] Lustre: server umount lustre-OST0000 complete [ 6814.691715] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6814.704042] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6814.734057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6814.739429] Lustre: Skipped 1 previous similar message [ 6818.744773] Lustre: server umount lustre-MDT0000 complete [ 6822.370317] LustreError: 159881:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6822.399895] LustreError: 159881:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6822.729523] LustreError: 157115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446296 with bad export cookie 806024051905665627 [ 6822.734100] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6822.740623] LustreError: 157115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6823.162109] Lustre: server umount lustre-MDT0001 complete [ 6826.944407] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 10:38:18 (1788446298) [ 6828.601769] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6830.350932] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 10:38:21 (1788446301) [ 6837.843747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6838.332062] LustreError: 161502:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6838.389676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6843.364534] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 6843.872079] LustreError: 161503:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6848.994965] LustreError: 161502:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6851.505616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6856.254963] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 6863.368681] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6869.196377] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6876.480629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6876.593461] LustreError: 163325:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6876.808880] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6876.818493] Lustre: Skipped 1 previous similar message [ 6883.116973] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 6890.495946] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6898.150604] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6898.165955] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6898.177638] Lustre: Skipped 1 previous similar message [ 6898.184908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6898.189894] Lustre: Skipped 2 previous similar messages [ 6902.810814] Lustre: server umount lustre-MDT0000 complete [ 6903.778524] LustreError: 161503:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6903.794464] LustreError: 161503:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6906.518703] LustreError: 161484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446380 with bad export cookie 806024051905666509 [ 6906.527160] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6906.547977] LustreError: 161484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6906.649115] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.57@tcp (stopping) [ 6906.658447] Lustre: Skipped 3 previous similar messages [ 6908.898884] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6908.924865] Lustre: Skipped 2 previous similar messages [ 6912.995885] Lustre: server umount lustre-MDT0001 complete [ 6920.473494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6932.865152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6933.138548] LustreError: 165344:0:(llog.c:1655:llog_backup()) MGC192.168.204.157@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6933.153054] Lustre: 165344:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.157@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6936.556801] LustreError: 165349:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6936.582695] LustreError: 165349:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6972.386235] LustreError: 165351:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6972.403341] LustreError: 165351:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 29 previous similar messages [ 6993.889317] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6993.903926] Lustre: 165344:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.157@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7034.873755] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7038.685440] LustreError: 165350:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7038.715327] LustreError: 165350:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 43 previous similar messages [ 7039.469811] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 7039.978794] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7047.474530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7051.233554] LustreError: 148556:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000bc28f6a4 x1875321527862784/t0(0) o250->MGC192.168.204.157@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7056.367548] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 7056.882920] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7064.496696] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7066.279104] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7072.250799] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7074.038615] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7092.707750] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7092.709207] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7092.710890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7092.710897] Lustre: Skipped 3 previous similar messages [ 7092.721968] Lustre: Skipped 2 previous similar messages [ 7097.496689] Lustre: server umount lustre-MDT0000 complete [ 7101.518976] LustreError: 161484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446575 with bad export cookie 806024051905672963 [ 7101.532427] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7101.540654] LustreError: 161484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7101.855526] Lustre: server umount lustre-MDT0001 complete [ 7116.556301] Lustre: server umount lustre-OST0000 complete [ 7136.318465] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [ 7144.426600] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 7176.089585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7176.371702] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7176.399113] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7176.476608] Lustre: lustre-MDT0000: new disk, initializing [ 7176.566695] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7176.571654] Lustre: Skipped 1 previous similar message [ 7176.586325] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7180.744280] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 7190.401695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7190.538749] Lustre: 170447:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7190.877689] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7195.580194] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 7205.164700] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7211.082135] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7218.100871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7218.393500] Lustre: lustre-OST0000: new disk, initializing [ 7218.396505] Lustre: Skipped 1 previous similar message [ 7218.400828] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7218.408808] Lustre: Skipped 2 previous similar messages [ 7218.414613] Lustre: 172416:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7225.585899] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 7226.965331] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7226.976440] Lustre: Skipped 1 previous similar message [ 7226.979256] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7227.168508] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7235.418075] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7240.463174] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7240.689756] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7245.874044] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7246.110426] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7252.970832] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7252.972362] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7252.978756] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7252.978764] Lustre: Skipped 2 previous similar messages [ 7252.993329] Lustre: Skipped 1 previous similar message [ 7254.439454] Lustre: server umount lustre-OST0000 complete [ 7264.406971] Lustre: server umount lustre-MDT0000 complete [ 7268.049686] LustreError: 170437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446741 with bad export cookie 806024051905689791 [ 7268.053325] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7268.067453] LustreError: 170437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7268.413925] Lustre: server umount lustre-MDT0001 complete [ 7278.614644] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 10:45:49 (1788446749) [ 7285.067174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7285.553397] LustreError: 174827:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7285.587773] LustreError: 174827:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 17 previous similar messages [ 7289.765470] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 7297.262138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7301.771810] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 7309.117087] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7314.524898] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7320.396304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7325.942856] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 7333.968694] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7343.522659] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 10:46:54 (1788446814) [ 7348.714714] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7348.726280] LustreError: Skipped 1 previous similar message [ 7348.734100] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7348.752820] Lustre: Skipped 2 previous similar messages [ 7348.758604] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7348.770420] Lustre: Skipped 5 previous similar messages [ 7354.053458] Lustre: server umount lustre-OST0000 complete [ 7372.255209] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7372.506210] Lustre: server umount lustre-MDT0000 complete [ 7376.124308] LustreError: 174807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446850 with bad export cookie 806024051905690673 [ 7376.137581] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7376.565872] Lustre: server umount lustre-MDT0001 complete [ 7386.991585] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 7390.236709] Key type lgssc unregistered [ 7390.535445] LNet: 178555:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7390.541377] LNetError: 178555:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7390.562576] LNet: Removed LNI 192.168.204.157@tcp [ 7391.464407] Key type .llcrypt unregistered [ 7391.470175] Key type ._llcrypt unregistered [ 7408.400262] Key type ._llcrypt registered [ 7408.402151] Key type .llcrypt registered [ 7408.511760] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 7409.393823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7409.407766] alg: No test for adler32 (adler32-zlib) [ 7410.466818] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 7410.673457] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 7412.327176] Key type lgssc registered [ 7413.442863] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7420.422955] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7420.448695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7435.999523] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7441.121898] LustreError: 179794:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.157@tcp: failed processing log, type 4: rc = -110 [ 7446.303868] LustreError: 179794:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7446.316346] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7446.321347] LustreError: MGC192.168.204.157@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7446.329365] LustreError: 179794:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7446.334909] LustreError: 179794:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -22 [ 7446.343158] LustreError: 179794:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 7446.389532] Lustre: server umount lustre-OST0000 complete [ 7446.396470] LustreError: 179794:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7454.299802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7455.792327] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7460.019920] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 7465.968914] LustreError: 180293:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7465.995222] LustreError: 180293:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7466.995959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7467.426817] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7471.839534] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 7479.216598] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7485.396418] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7492.875770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7493.056704] LustreError: 182110:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7493.072541] Lustre: 182110:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.204.157@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7493.112850] LustreError: 182115:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7493.278023] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7499.777264] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 7505.691437] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7508.449179] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7508.462216] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7508.472335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7511.015511] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7511.015817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7511.031216] Lustre: Skipped 2 previous similar messages [ 7513.782951] Lustre: server umount lustre-MDT0000 complete [ 7516.133615] LustreError: 180293:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7516.171631] LustreError: 180293:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7517.150340] LustreError: 180273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446991 with bad export cookie 16232359224065365287 [ 7517.157155] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7517.164071] LustreError: 180273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7517.443084] Lustre: server umount lustre-MDT0001 complete [ 7531.140731] Lustre: server umount lustre-OST0000 complete [ 7539.832758] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [ 7549.914220] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 7582.877257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7583.138904] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7583.197443] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7583.302306] Lustre: lustre-MDT0000: new disk, initializing [ 7583.404241] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7583.427307] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7587.479138] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 7596.882189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7597.010616] Lustre: 185635:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7597.080920] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7597.088465] Lustre: Skipped 1 previous similar message [ 7597.188738] Lustre: lustre-MDT0001: new disk, initializing [ 7597.283457] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7597.327930] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7597.341443] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7601.242344] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 7609.924965] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7614.726883] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7620.345568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7620.773538] Lustre: lustre-OST0000: new disk, initializing [ 7620.778310] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7620.796946] Lustre: 187601:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7620.901492] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7626.292222] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 7628.332879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7628.340316] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7628.460487] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7636.526674] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7640.893406] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7641.185336] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7646.025517] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7646.228536] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7649.273267] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7649.275131] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7649.283277] LustreError: Skipped 1 previous similar message [ 7649.285438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7649.296130] Lustre: Skipped 2 previous similar messages [ 7649.319250] Lustre: Skipped 1 previous similar message [ 7654.408400] Lustre: server umount lustre-OST0000 complete [ 7658.989398] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7659.003575] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7659.036843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7664.562394] Lustre: server umount lustre-MDT0000 complete [ 7664.612830] LustreError: 185646:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7664.628966] LustreError: 185646:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7668.298601] LustreError: 185626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447142 with bad export cookie 16232359224065366008 [ 7668.299901] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7668.306203] LustreError: 185626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7668.639604] Lustre: server umount lustre-MDT0001 complete [ 7676.947647] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 10:52:28 (1788447148) [ 7693.831353] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 7696.807212] Key type lgssc unregistered [ 7697.151104] LNet: 190543:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7697.158624] LNetError: 190543:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7697.183828] LNet: Removed LNI 192.168.204.157@tcp [ 7697.880286] Key type .llcrypt unregistered [ 7697.893966] Key type ._llcrypt unregistered [ 7716.351706] Key type ._llcrypt registered [ 7716.356612] Key type .llcrypt registered [ 7716.458807] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 7717.471608] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7717.529644] alg: No test for adler32 (adler32-zlib) [ 7718.633584] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 7718.963750] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 7720.693981] Key type lgssc registered [ 7721.781846] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7730.416446] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7730.440050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7731.878943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7736.237243] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 7741.413258] LustreError: 191759:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7741.437735] LustreError: 191759:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7745.200823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7745.625130] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7750.187598] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 7757.690994] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7763.168942] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7770.187349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7770.402938] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7770.406860] LustreError: 193623:0:(llog.c:1696:llog_backup()) MGC192.168.204.157@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7770.413188] Lustre: 193623:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.204.157@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7770.521569] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7776.180487] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 7783.139893] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7786.465092] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7786.474794] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7786.484417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7786.981539] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7787.006983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7787.017569] Lustre: Skipped 1 previous similar message [ 7791.404994] Lustre: server umount lustre-MDT0000 complete [ 7792.102529] LustreError: 191758:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7792.124755] LustreError: 191758:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7794.995158] LustreError: 191739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447268 with bad export cookie 15080742725346745109 [ 7794.997675] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7795.002678] LustreError: 191739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7795.339565] Lustre: server umount lustre-MDT0001 complete [ 7808.317031] Lustre: server umount lustre-OST0000 complete [ 7816.824321] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [ 7826.704927] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 7857.712796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7857.896269] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7857.921099] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7857.989154] Lustre: lustre-MDT0000: new disk, initializing [ 7858.043916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7858.060746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7862.229881] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 7871.283310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7871.374776] Lustre: 197145:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7871.428677] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7871.431361] Lustre: Skipped 1 previous similar message [ 7871.434812] LustreError: 197929:0:(llog.c:1696:llog_backup()) MGC192.168.204.157@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7871.443883] LustreError: 197929:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7871.449406] Lustre: 197929:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.157@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7871.481828] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7871.490957] Lustre: Skipped 1 previous similar message [ 7871.547955] Lustre: lustre-MDT0001: new disk, initializing [ 7871.630976] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7871.664758] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7871.672946] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7876.180504] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 7885.040507] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7889.604270] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7895.697985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7895.935163] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7895.937044] Lustre: Skipped 1 previous similar message [ 7895.947490] LustreError: 199090:0:(llog.c:1696:llog_backup()) MGC192.168.204.157@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7895.962426] LustreError: 199090:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7895.970297] Lustre: 199090:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.157@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7895.978022] Lustre: 199090:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7896.009933] Lustre: lustre-OST0000: new disk, initializing [ 7896.017416] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7896.025753] Lustre: 199114:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7896.151403] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7902.080619] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 7903.269819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7903.284974] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7903.390943] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7910.304635] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7914.394469] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7914.619621] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7919.166621] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7919.398874] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7923.684893] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7923.694641] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7923.714657] Lustre: Skipped 1 previous similar message [ 7923.720347] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7927.546144] Lustre: server umount lustre-OST0000 complete [ 7933.414320] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7933.426461] LustreError: Skipped 1 previous similar message [ 7933.435921] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7933.452545] Lustre: Skipped 1 previous similar message [ 7933.462922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7933.470566] Lustre: Skipped 2 previous similar messages [ 7939.556987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7939.563225] Lustre: Skipped 3 previous similar messages [ 7945.183134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7945.408761] Lustre: server umount lustre-MDT0000 complete [ 7948.934604] LustreError: 197140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447422 with bad export cookie 15080742725346745823 [ 7948.944046] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7948.946921] LustreError: 197140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7949.223483] Lustre: server umount lustre-MDT0001 complete [ 7957.753044] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 10:57:09 (1788447429) [ 7976.590393] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 7988.173994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7988.422390] LustreError: 202241:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7988.463618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7991.923531] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 7993.825255] LustreError: 202242:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7998.946372] LustreError: 202241:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8001.479239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8007.502790] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 8010.758902] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8017.940746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8024.356493] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 8032.970057] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8033.313924] Lustre: lustre-OST0001: new disk, initializing [ 8033.324485] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8033.344654] Lustre: 204549:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8033.639618] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8033.647608] Lustre: Skipped 2 previous similar messages [ 8041.066913] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 8041.135714] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8041.146744] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8041.236698] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8053.456495] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8060.166435] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8062.219730] Lustre: DEBUG MARKER: ADD OST3 [ 8066.632261] loop: module loaded [ 8066.700470] loop0: detected capacity change from 0 to 204800000 [ 8066.844776] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8066.852272] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8066.866644] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8072.449968] loop0: detected capacity change from 0 to 204800000 [ 8073.475223] Lustre: DEBUG MARKER: STOP OST3 [ 8077.100177] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8077.284514] Lustre: lustre-OST0003: new disk, initializing [ 8077.292236] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8077.297311] Lustre: 206529:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8078.994115] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8079.086196] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8079.094874] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8079.127232] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8080.546397] Lustre: Failing over lustre-OST0003 [ 8080.689644] Lustre: server umount lustre-OST0003 complete [ 8081.058292] LustreError: 206769:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8083.080684] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 8084.246568] LustreError: 203728:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.204.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8084.467330] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8084.472420] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8084.488372] Lustre: Skipped 1 previous similar message [ 8093.554145] Lustre: DEBUG MARKER: START OST3 again [ 8100.818525] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8101.108834] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8101.112054] Lustre: Skipped 1 previous similar message [ 8101.127195] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8102.196299] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8102.819061] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8102.819700] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8106.033255] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 8125.651516] Lustre: Failing over lustre-OST0003 [ 8125.748253] Lustre: server umount lustre-OST0003 complete [ 8126.965175] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8126.981084] Lustre: Skipped 1 previous similar message [ 8126.991582] LustreError: 204914:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8127.003105] LustreError: 204914:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 8142.323601] LustreError: 191067:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff98188e63fb80 x1875322802999296/t0(0) o8->lustre-OST0003-osc-MDT0000@0@lo:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8142.362209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8142.375580] Lustre: Skipped 5 previous similar messages [ 8148.722873] Lustre: server umount lustre-MDT0000 complete [ 8151.810130] LustreError: 202224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447625 with bad export cookie 15080742725346746705 [ 8151.817548] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8151.832167] LustreError: 202224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8152.544251] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8152.564116] Lustre: Skipped 5 previous similar messages [ 8161.702538] LustreError: 202238:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8161.708521] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8161.723742] LustreError: 202238:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 8161.754152] Lustre: Skipped 9 previous similar messages [ 8166.367148] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8166.848971] Lustre: server umount lustre-MDT0001 complete [ 8171.584861] Lustre: server umount lustre-OST0000 complete [ 8175.776553] Lustre: server umount lustre-OST0001 complete [ 8182.468612] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [ 8191.757702] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 8229.345255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8229.610190] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8229.631926] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8229.706327] Lustre: lustre-MDT0000: new disk, initializing [ 8229.799243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8229.831628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8233.431303] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 8242.734216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8242.848337] Lustre: 212020:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8243.119909] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8247.061313] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 8255.339896] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8261.071189] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8267.425311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8267.725798] Lustre: lustre-OST0000: new disk, initializing [ 8267.730034] Lustre: Skipped 1 previous similar message [ 8267.734884] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8267.740800] Lustre: Skipped 2 previous similar messages [ 8267.749276] Lustre: 213986:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8269.194968] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8269.202839] Lustre: Skipped 1 previous similar message [ 8269.209471] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8269.303752] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8273.985559] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 8283.346529] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8288.418545] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8288.694575] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8293.909730] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8294.223279] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8300.017766] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8300.019473] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8300.030030] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8300.044600] Lustre: Skipped 2 previous similar messages [ 8300.072407] LustreError: Skipped 1 previous similar message [ 8310.239181] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8310.330494] Lustre: server umount lustre-OST0000 complete [ 8319.451048] Lustre: server umount lustre-MDT0000 complete [ 8323.552298] LustreError: 212030:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8323.573233] LustreError: 212030:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8323.676083] LustreError: 212010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447797 with bad export cookie 15080742725346749008 [ 8323.685976] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8324.120959] Lustre: server umount lustre-MDT0001 complete [ 8328.651273] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 11:03:19 (1788447799) [ 8345.769856] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [ 8357.706320] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 8390.404425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8390.647824] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8390.678922] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8390.766594] Lustre: lustre-MDT0000: new disk, initializing [ 8390.865546] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8394.524185] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 8403.353597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8403.450915] Lustre: 218600:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8403.665395] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8407.356499] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 8416.073600] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8420.914758] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8427.015172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8427.290242] Lustre: 220568:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8429.465276] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8433.344861] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 8442.990180] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8446.901540] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8447.103374] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8451.481116] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8451.690349] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8455.146403] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8455.148216] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8455.151284] LustreError: Skipped 1 previous similar message [ 8455.152988] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8455.166094] Lustre: Skipped 3 previous similar messages [ 8455.168266] Lustre: Skipped 9 previous similar messages [ 8459.461574] Lustre: server umount lustre-OST0000 complete [ 8477.151226] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8477.241576] Lustre: server umount lustre-MDT0000 complete [ 8480.728869] LustreError: 218592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447954 with bad export cookie 15080742725346749890 [ 8480.739296] LustreError: 218612:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8480.757641] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8480.770845] LustreError: 218592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8481.007940] Lustre: server umount lustre-MDT0001 complete [ 8489.568455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8490.252330] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8490.260431] Lustre: Skipped 5 previous similar messages [ 8494.387317] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 8502.280829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8508.642834] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 8518.159923] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8524.851573] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8532.319372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8539.200092] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 8546.086736] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8556.516039] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8556.520165] LustreError: Skipped 1 previous similar message [ 8562.213746] Lustre: server umount lustre-OST0000 complete [ 8572.045864] Lustre: server umount lustre-MDT0000 complete [ 8575.574861] LustreError: 222767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448049 with bad export cookie 15080742725346750772 [ 8575.580332] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8575.584059] LustreError: 222767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8575.861873] Lustre: server umount lustre-MDT0001 complete [ 8684.185870] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 8687.115429] Key type lgssc unregistered [ 8687.558554] LNet: 226447:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8687.573912] LNetError: 226447:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8687.592968] LNet: Removed LNI 192.168.204.157@tcp [ 8688.876350] Key type .llcrypt unregistered [ 8688.878577] Key type ._llcrypt unregistered [ 8723.766653] Key type ._llcrypt registered [ 8723.767994] Key type .llcrypt registered [ 8723.936887] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [ 8739.136624] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 8739.989089] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8740.022110] alg: No test for adler32 (adler32-zlib) [ 8741.138101] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8741.330489] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 8743.039928] Key type lgssc registered [ 8744.338213] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8780.901889] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8780.915701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8782.159295] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8782.218304] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8782.268423] Lustre: lustre-MDT0000: new disk, initializing [ 8782.321201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8782.333503] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8787.621402] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 8796.262122] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8796.359602] Lustre: 229802:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8796.408084] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8796.411911] Lustre: Skipped 1 previous similar message [ 8796.486960] Lustre: lustre-MDT0001: new disk, initializing [ 8796.591843] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8796.622172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8796.631546] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8800.294526] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 8810.327382] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8816.977412] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8824.308185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8824.660601] Lustre: lustre-OST0000: new disk, initializing [ 8824.664244] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8824.671837] Lustre: 231770:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8824.781274] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8829.969327] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 8834.621109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8834.628102] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8834.770040] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8837.286511] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8841.220826] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8841.415809] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8845.928571] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8846.206492] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8850.406034] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8850.407141] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8850.408177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8850.415204] Lustre: Skipped 1 previous similar message [ 8850.420236] LustreError: Skipped 1 previous similar message [ 8854.411384] Lustre: server umount lustre-OST0000 complete [ 8860.653162] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8860.658294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8860.672766] Lustre: Skipped 1 previous similar message [ 8860.689928] Lustre: Skipped 2 previous similar messages [ 8864.515094] Lustre: server umount lustre-MDT0000 complete [ 8868.435677] LustreError: 229792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448342 with bad export cookie 3630495690167255904 [ 8868.441396] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8868.454384] LustreError: 229792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8868.741766] Lustre: server umount lustre-MDT0001 complete [ 8888.516397] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 8897.546366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8897.919279] LustreError: 234704:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8897.935196] LustreError: 234704:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8898.000097] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8902.307415] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 8903.137683] LustreError: 234705:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8907.238699] LustreError: 234704:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8911.174884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8911.519289] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8915.858251] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 8918.798310] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8925.341092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8925.669060] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8933.315841] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 8942.774329] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8942.965407] Lustre: lustre-OST0001: new disk, initializing [ 8942.971982] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8942.982838] Lustre: 237012:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8943.139412] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8943.879374] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8943.896900] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8943.956858] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8949.787285] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 8960.232431] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8964.805794] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8977.494290] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 11:14:08 (1788448448) [ 8980.962043] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8980.982083] Lustre: Skipped 2 previous similar messages [ 8980.988367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8980.993926] Lustre: Skipped 1 previous similar message [ 8985.772077] Lustre: server umount lustre-MDT0000 complete [ 8986.090073] LustreError: 236445:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8986.121851] LustreError: 236445:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8989.367374] LustreError: 234684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448463 with bad export cookie 3630495690167256786 [ 8989.376403] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8989.385668] LustreError: 234684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8989.872633] Lustre: server umount lustre-MDT0001 complete [ 9003.697472] Lustre: server umount lustre-OST0000 complete [ 9017.870905] Lustre: server umount lustre-OST0001 complete [ 9024.305716] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [ 9033.595961] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 9074.647513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9075.033417] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9075.068694] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9075.220313] Lustre: lustre-MDT0000: new disk, initializing [ 9075.407818] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9075.442566] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9080.623689] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 9091.677314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9091.804776] Lustre: 240961:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9091.893719] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9091.904420] Lustre: Skipped 1 previous similar message [ 9092.067316] Lustre: lustre-MDT0001: new disk, initializing [ 9092.269059] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9092.295930] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9097.892344] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 9108.821248] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9115.458642] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9123.320624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9123.712532] Lustre: lustre-OST0000: new disk, initializing [ 9123.716162] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9123.723349] Lustre: 242931:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9124.901650] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9124.917252] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9125.058300] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9131.095923] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 9141.319951] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9147.403692] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9147.626527] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9153.125850] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9153.433939] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9156.068873] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9156.083148] Lustre: Skipped 2 previous similar messages [ 9156.083966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9156.088952] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9156.099690] Lustre: Skipped 3 previous similar messages [ 9161.191420] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9161.199718] Lustre: Skipped 1 previous similar message [ 9161.862870] Lustre: server umount lustre-OST0000 complete [ 9166.305386] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9171.424357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9171.434391] Lustre: Skipped 3 previous similar messages [ 9172.413169] Lustre: server umount lustre-MDT0000 complete [ 9176.440698] LustreError: 240951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448650 with bad export cookie 3630495690167258921 [ 9176.446909] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9176.453386] LustreError: 240951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9176.847805] Lustre: server umount lustre-MDT0001 complete [ 9185.967777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9186.593783] LustreError: 245146:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9186.652243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9186.657847] Lustre: Skipped 2 previous similar messages [ 9191.180481] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 9197.024979] LustreError: 245147:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9197.048429] LustreError: 245147:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9199.714568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9204.569927] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 9213.114549] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9219.821513] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9227.174643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9233.490490] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 9240.610283] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9388.519092] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9388.520118] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9388.531397] Lustre: Skipped 1 previous similar message [ 9388.539062] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9388.545404] Lustre: Skipped 1 previous similar message [ 9393.590251] Lustre: server umount lustre-OST0000 complete [ 9406.212888] Lustre: server umount lustre-MDT0000 complete [ 9408.992970] LustreError: 245141:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9409.008654] LustreError: 245141:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9410.473430] LustreError: 245128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448884 with bad export cookie 3630495690167259796 [ 9410.478697] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9410.487858] LustreError: 245128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9411.013805] Lustre: server umount lustre-MDT0001 complete [ 9506.678534] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 9509.116568] Key type lgssc unregistered [ 9509.376471] LNet: 248754:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9509.399758] LNetError: 248754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9509.435595] LNet: Removed LNI 192.168.204.157@tcp [ 9510.245170] Key type .llcrypt unregistered [ 9510.248436] Key type ._llcrypt unregistered [ 9536.435631] Key type ._llcrypt registered [ 9536.439309] Key type .llcrypt registered [ 9536.554472] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [ 9552.777810] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 9553.693440] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9553.872805] alg: No test for adler32 (adler32-zlib) [ 9554.939227] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 9555.173584] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 9556.887204] Key type lgssc registered [ 9557.973873] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9603.758756] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 9615.442806] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9615.468498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9616.912535] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9616.942250] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9617.040849] Lustre: lustre-MDT0000: new disk, initializing [ 9617.118336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9617.147213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9622.523746] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 9633.999827] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9634.161248] Lustre: 252825:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9634.225949] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9634.229282] Lustre: Skipped 1 previous similar message [ 9634.322907] Lustre: lustre-MDT0001: new disk, initializing [ 9634.409916] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9634.452594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9634.474284] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9639.609947] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 9645.093448] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9651.895905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9652.244205] Lustre: lustre-OST0000: new disk, initializing [ 9652.248542] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9652.263600] Lustre: 254456:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9652.415937] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9657.924771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9657.934244] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9658.095804] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9658.987670] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 9670.480929] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9670.721338] Lustre: lustre-OST0001: new disk, initializing [ 9670.728483] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9670.738333] Lustre: 255326:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9670.857239] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9677.793316] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 9679.507464] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9679.527663] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9679.698093] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9688.827759] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9694.818327] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9710.574523] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9710.586660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9710.589622] Lustre: Skipped 1 previous similar message [ 9710.610528] Lustre: Skipped 2 previous similar messages [ 9714.059540] Lustre: server umount lustre-MDT0000 complete [ 9718.310707] LustreError: 252816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449192 with bad export cookie 4155224701024858837 [ 9718.319923] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9718.334111] LustreError: 252816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9718.867296] Lustre: server umount lustre-MDT0001 complete [ 9732.409253] Lustre: server umount lustre-OST0000 complete [ 9746.898805] Lustre: server umount lustre-OST0001 complete [ 9760.667259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9770.267379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9778.157214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9787.393346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9807.765788] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 9819.944401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9819.962151] Lustre: lustre-MDT0000: reset Object Index mappings [ 9820.234296] LustreError: 260336:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9820.250382] LustreError: 260336:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9820.287661] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9825.490669] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 9825.761479] LustreError: 260337:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9830.884378] LustreError: 260336:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9835.011416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9835.035393] Lustre: lustre-MDT0001: reset Object Index mappings [ 9835.422816] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9840.388930] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 9843.990480] Lustre: 261477:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9851.990953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9852.554116] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9859.806529] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 9860.720075] LustreError: 261833:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9865.843731] LustreError: 262228:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9865.854462] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9865.874177] LustreError: 262228:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9869.509127] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9874.941397] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9876.248204] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 9883.565143] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9887.279070] Lustre: 263350:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9894.452841] Lustre: server umount lustre-MDT0000 complete [ 9895.402933] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9895.408465] LustreError: 260332:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9895.427720] Lustre: Skipped 4 previous similar messages [ 9895.486982] LustreError: 260332:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9899.176789] LustreError: 260318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449373 with bad export cookie 4155224701024865627 [ 9899.216705] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9899.803940] Lustre: server umount lustre-MDT0001 complete [ 9914.391134] Lustre: server umount lustre-OST0000 complete [ 9916.384856] Lustre: 250327:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788449374/real 1788449374] req@ffff9819b71e3480 x1875324729159936/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788449390 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9916.404286] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9916.415958] Lustre: Skipped 1 previous similar message [ 9918.620401] Lustre: server umount lustre-OST0001 complete [ 9937.267503] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 11:30:08 (1788449408) [ 9952.118930] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [ 9962.377449] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 9998.101962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9998.460190] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9998.529727] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9998.692796] Lustre: lustre-MDT0000: new disk, initializing [ 9998.861625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9998.873323] Lustre: Skipped 1 previous similar message [ 9998.921124] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10004.007462] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [10014.982601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10015.129981] Lustre: 267602:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10015.203491] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10015.208097] Lustre: Skipped 1 previous similar message [10015.324677] Lustre: lustre-MDT0001: new disk, initializing [10015.535505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10015.546243] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10019.408958] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [10029.582609] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10035.577976] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10042.415372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10042.871662] Lustre: lustre-OST0000: new disk, initializing [10042.877071] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10042.886917] Lustre: 269570:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10044.169567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10044.181455] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10044.398364] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10050.165322] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [10058.649297] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10063.069628] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10063.257942] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10068.040346] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10068.258939] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10075.110286] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10075.120389] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10075.131747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10075.145942] Lustre: Skipped 1 previous similar message [10075.163991] Lustre: Skipped 2 previous similar messages [10080.236495] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10080.243859] Lustre: Skipped 1 previous similar message [10084.320738] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10084.635194] Lustre: server umount lustre-OST0000 complete [10090.466436] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10090.473437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10090.523691] Lustre: Skipped 1 previous similar message [10095.393841] Lustre: server umount lustre-MDT0000 complete [10095.591047] LustreError: 269580:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10095.611648] LustreError: 269580:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10099.582462] LustreError: 267594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449573 with bad export cookie 4155224701024867713 [10099.586463] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10099.596916] LustreError: 267594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10100.019737] Lustre: server umount lustre-MDT0001 complete [10118.396662] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [10127.737969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10128.121255] LustreError: 272503:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10128.190738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10128.197206] Lustre: Skipped 2 previous similar messages [10133.135083] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [10141.946588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10147.032211] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [10150.368884] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10157.112755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10164.155980] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [10171.702851] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10171.848317] Lustre: lustre-OST0001: new disk, initializing [10171.851641] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10171.862792] Lustre: 274812:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10173.759729] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10173.775105] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10173.872809] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10177.443723] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [10187.855382] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10192.845304] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10200.597149] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10200.622307] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10214.376698] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10214.386508] LustreError: Skipped 1 previous similar message [10214.391549] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10214.421940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10214.424362] Lustre: Skipped 1 previous similar message [10219.897902] Lustre: server umount lustre-MDT0000 complete [10221.034827] LustreError: 272503:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10221.052536] LustreError: 272503:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10224.142418] LustreError: 274012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449698 with bad export cookie 4155224701024868595 [10224.155915] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10224.162102] LustreError: 274012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10224.633190] Lustre: server umount lustre-MDT0001 complete [10238.901449] Lustre: server umount lustre-OST0000 complete [10251.535854] Lustre: server umount lustre-OST0001 complete [10268.313510] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [10277.676120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10278.020526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10278.024281] Lustre: Skipped 3 previous similar messages [10281.955379] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [10289.915630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10294.786539] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [10297.877186] Lustre: 279302:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10304.795398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10310.734461] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [10318.543196] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10323.124797] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10323.158866] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10326.862708] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [10335.356546] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10340.023256] Lustre: 281182:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10345.682531] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10345.719406] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10357.916984] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 11:37:09 (1788449829) [10374.632166] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10374.639403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10374.655555] Lustre: Skipped 5 previous similar messages [10374.667861] Lustre: Skipped 7 previous similar messages [10378.679499] Lustre: server umount lustre-MDT0000 complete [10379.748677] LustreError: 278158:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10379.772400] LustreError: 278158:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [10381.941436] LustreError: 279305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449855 with bad export cookie 4155224701024875462 [10381.952140] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10381.956194] LustreError: 279305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10388.383640] Lustre: server umount lustre-MDT0001 complete [10406.369353] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10406.480033] Lustre: server umount lustre-OST0000 complete [10410.371496] Lustre: server umount lustre-OST0001 complete [10417.153447] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [10427.015017] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [10470.947425] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [10480.687184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10480.962466] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10481.005539] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10481.124945] Lustre: lustre-MDT0000: new disk, initializing [10481.282707] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10486.306486] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [10496.939031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10497.072624] Lustre: 285705:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10497.324191] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10501.557265] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [10512.241179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10512.512905] Lustre: 287340:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10514.023335] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10517.831577] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [10527.805636] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10527.960428] Lustre: 288213:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10529.339714] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10529.351059] Lustre: Skipped 1 previous similar message [10529.397909] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10534.094740] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [10544.105524] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10548.585904] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10548.588221] Lustre: Skipped 1 previous similar message [10564.148226] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 11:40:35 (1788450035) [10569.185066] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10569.194303] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10569.204607] Lustre: Skipped 3 previous similar messages [10569.216464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10569.227590] Lustre: Skipped 4 previous similar messages [10572.659701] Lustre: server umount lustre-MDT0000 complete [10576.903733] LustreError: 285698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450050 with bad export cookie 4155224701024882959 [10576.905413] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10576.920809] LustreError: 285698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10577.396401] Lustre: server umount lustre-MDT0001 complete [10591.807554] Lustre: server umount lustre-OST0000 complete [10606.129787] Lustre: server umount lustre-OST0001 complete [10613.937720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10614.477333] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10614.484793] Lustre: Skipped 7 previous similar messages [10618.618463] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [10627.188152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10632.782510] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [10641.019951] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10647.065119] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10654.273984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10659.832876] LustreError: 292285:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10659.862664] LustreError: 292285:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [10661.637189] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [10669.984261] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10691.210956] loop1: detected capacity change from 0 to 204800000 [10691.428649] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10691.462097] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10691.482981] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10697.344810] loop1: detected capacity change from 0 to 204800000 [10703.902307] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10704.015619] LustreError: 290447:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs32022: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10704.032913] LustreError: 290447:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs32022 [10704.036199] LustreError: 290447:0:(mgs_handler.c:517:mgs_target_reg()) fs32022-MDT0001: Error checking target index: rc = -13 [10704.055790] LustreError: fs32022-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10704.073567] LustreError: 294177:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -13 [10704.085952] LustreError: 294177:0:(tgt_mount.c:2132:server_put_super()) no obd fs32022-MDT0001 [10704.089976] LustreError: 294177:0:(tgt_mount.c:116:server_deregister_mount()) fs32022-MDT0001 not registered [10704.166056] Lustre: server umount fs32022-MDT0001 complete [10704.169842] LustreError: 294177:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10714.358278] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10714.743459] Lustre: Setting parameter fs32022-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs32022-MDT0001 [10714.795284] Lustre: srv-fs32022-MDT0001: No data found on store. Initialize space. [10714.799978] Lustre: Skipped 4 previous similar messages [10714.916475] Lustre: fs32022-MDT0001: new disk, initializing [10714.919976] Lustre: Skipped 3 previous similar messages [10715.078947] Lustre: srv-fs32022-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10715.096911] Lustre: 294644:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs32022-MDT0001-mdtlov: can't create new llog: rc = -115 [10715.109320] LustreError: 294644:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs32022-MDT0001-osd: get update log duration 1, retries 0, failed: rc = -115 [10720.388995] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [10727.631848] Lustre: server umount fs32022-MDT0001 complete [10739.698267] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10741.688681] loop1: detected capacity change from 0 to 204800000 [10748.655961] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10749.106354] Lustre: 296211:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs32022-MDT0001-mdtlov: can't create new llog: rc = -115 [10749.125260] LustreError: 296211:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs32022-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10754.456562] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [10755.060711] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10760.319962] Lustre: server umount fs32022-MDT0001 complete [10770.912557] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10770.926721] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10770.939935] Lustre: Skipped 3 previous similar messages [10770.950927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10770.956820] Lustre: Skipped 4 previous similar messages [10785.247133] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10785.552062] Lustre: server umount lustre-MDT0000 complete [10793.598145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10793.728453] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10798.211023] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [10799.114708] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10806.577913] loop1: detected capacity change from 0 to 204800000 [10812.935780] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10813.325318] Lustre: 298661:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs32022-MDT0001-mdtlov: can't create new llog: rc = -115 [10813.334815] LustreError: 298661:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs32022-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10819.095591] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [10823.934215] Lustre: server umount fs32022-MDT0001 complete [10847.202108] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10847.216039] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10847.223216] Lustre: Skipped 9 previous similar messages [10851.718962] Lustre: server umount lustre-OST0000 complete [10861.915739] Lustre: server umount lustre-MDT0000 complete [10865.913361] LustreError: 299154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450339 with bad export cookie 4155224701024885640 [10865.916566] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10865.928498] LustreError: 299154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10866.294917] Lustre: server umount lustre-MDT0001 complete [10875.977919] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [10878.935623] Key type lgssc unregistered [10879.184602] LNet: 301006:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10879.197334] LNetError: 301006:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10879.231162] LNet: Removed LNI 192.168.204.157@tcp [10880.165157] Key type .llcrypt unregistered [10880.167103] Key type ._llcrypt unregistered [10901.427337] Key type ._llcrypt registered [10901.429181] Key type .llcrypt registered [10901.529202] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [10903.087224] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10903.128615] alg: No test for adler32 (adler32-zlib) [10904.216651] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [10904.459248] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [10906.327644] Key type lgssc registered [10907.585650] Lustre: Echo OBD driver; http://www.lustre.org/ [10916.738403] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10916.787665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10918.595814] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10923.265550] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [10928.099267] LustreError: 302275:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10928.120825] LustreError: 302275:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10932.948882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10933.341533] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10937.948595] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [10944.909954] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10949.942912] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10955.616531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10955.977129] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10959.018924] LustreError: 304104:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10961.623617] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [10963.114879] LustreError: 304106:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10963.128925] LustreError: 304106:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10968.549798] LustreError: 304106:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10968.567890] LustreError: 304106:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10969.088528] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10973.666906] LustreError: 304106:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10973.696932] LustreError: 304106:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10983.913188] LustreError: 304106:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10983.941200] LustreError: 304106:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10985.444198] loop1: detected capacity change from 0 to 204800000 [10985.730912] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10985.750987] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10985.767750] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10991.042442] loop1: detected capacity change from 0 to 204800000 [10996.153128] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10996.287851] Lustre: fs32416-OST0000: new disk, initializing [10996.290832] Lustre: srv-fs32416-OST0000: No data found on store. Initialize space. [10996.299829] Lustre: 306011:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10996.393355] Lustre: fs32416-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11001.575839] LustreError: 304103:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11001.597687] LustreError: 304103:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [11002.950830] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [11011.711665] Lustre: server umount fs32416-OST0000 complete [11027.821281] loop1: detected capacity change from 0 to 204800000 [11033.763949] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11034.050592] Lustre: fs32416-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11035.114477] LustreError: 304104:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11035.150371] LustreError: 304104:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 19 previous similar messages [11041.006778] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [11045.359377] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11049.744119] loop2: detected capacity change from 0 to 204800000 [11050.035239] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11050.051353] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11050.079225] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11055.399429] loop2: detected capacity change from 0 to 204800000 [11060.806488] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11060.875917] LustreError: fs32416-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11060.895340] LustreError: fs32416-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11060.904271] LustreError: 309050:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -13 [11060.915160] LustreError: 309050:0:(tgt_mount.c:2132:server_put_super()) no obd fs32416-OST0001 [11060.920151] LustreError: 309050:0:(tgt_mount.c:116:server_deregister_mount()) fs32416-OST0001 not registered [11060.942643] Lustre: server umount fs32416-OST0001 complete [11060.949608] LustreError: 309050:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11070.867996] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11071.120197] Lustre: fs32416-OST0001: new disk, initializing [11071.126794] Lustre: srv-fs32416-OST0001: No data found on store. Initialize space. [11071.133304] Lustre: 309600:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11071.237856] Lustre: fs32416-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11077.836539] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [11086.555662] Lustre: server umount fs32416-OST0000 complete [11095.815762] Lustre: server umount fs32416-OST0001 complete [11101.683824] LustreError: 304103:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11101.704292] LustreError: 304103:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [11108.804947] loop1: detected capacity change from 0 to 204800000 [11114.881203] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11115.142432] Lustre: fs32416-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11122.044833] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [11128.027572] loop2: detected capacity change from 0 to 204800000 [11133.996968] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11141.496644] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [11150.708779] Lustre: server umount fs32416-OST0000 complete [11159.851251] Lustre: server umount fs32416-OST0001 complete [11173.352366] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11173.359640] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11173.366095] LustreError: Skipped 1 previous similar message [11173.367940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11173.387844] Lustre: Skipped 1 previous similar message [11178.429442] Lustre: server umount lustre-OST0000 complete [11183.594322] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11183.619680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11183.629171] Lustre: Skipped 2 previous similar messages [11188.729350] Lustre: server umount lustre-MDT0000 complete [11192.407330] LustreError: 302257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450666 with bad export cookie 2624802606595710635 [11192.411745] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11192.414476] LustreError: 302257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11192.769650] Lustre: server umount lustre-MDT0001 complete [11205.842745] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [11208.999235] Key type lgssc unregistered [11209.252481] LNet: 315523:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11209.260656] LNetError: 315523:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11209.278776] LNet: Removed LNI 192.168.204.157@tcp [11209.965272] Key type .llcrypt unregistered [11209.967072] Key type ._llcrypt unregistered [11244.032044] Key type ._llcrypt registered [11244.035554] Key type .llcrypt registered [11244.095906] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [11260.277613] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [11261.351127] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11261.382126] alg: No test for adler32 (adler32-zlib) [11262.494303] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [11262.807941] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [11264.495247] Key type lgssc registered [11266.002748] Lustre: Echo OBD driver; http://www.lustre.org/ [11304.852062] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11304.875390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11306.129894] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11306.173815] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11306.251473] Lustre: lustre-MDT0000: new disk, initializing [11306.420830] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11306.447785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11311.213872] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [11321.942358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11322.165080] Lustre: 319380:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11322.279661] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11322.283726] Lustre: Skipped 1 previous similar message [11322.423166] Lustre: lustre-MDT0001: new disk, initializing [11322.526085] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11322.569460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11322.583625] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11327.631267] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [11336.625784] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11341.706663] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11346.538323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11346.726412] Lustre: lustre-OST0000: new disk, initializing [11346.730201] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11346.736049] Lustre: 321349:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11346.807730] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11351.645782] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [11356.211778] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11356.224811] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11356.307605] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11359.783442] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11361.706479] Lustre: Mounted lustre-client - version 2.17.58_2_gee80f9b [11362.718401] Lustre: Unmounted lustre-client [11372.000803] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11372.010115] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11372.019413] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11372.025524] Lustre: Skipped 1 previous similar message [11377.136558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11377.157742] Lustre: Skipped 2 previous similar messages [11377.729277] Lustre: server umount lustre-OST0000 complete [11382.249259] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11382.252334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11382.262817] Lustre: Skipped 1 previous similar message [11387.360739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11387.364745] Lustre: Skipped 2 previous similar messages [11388.470322] Lustre: server umount lustre-MDT0000 complete [11392.230303] LustreError: 319372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450866 with bad export cookie 18250513535670522513 [11392.240422] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11392.252905] LustreError: 319372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11392.791100] Lustre: server umount lustre-MDT0001 complete [11405.188440] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [11408.054931] Key type lgssc unregistered [11419.551332] LNet: 1 local NIs in recovery (showing 1): 192.168.204.157@tcp [11419.616370] LNet: 323510:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11419.632215] LNetError: 323510:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11490.346619] LNet: Removed LNI 192.168.204.157@tcp [11491.189185] Key type .llcrypt unregistered [11491.190590] Key type ._llcrypt unregistered [11507.717708] Key type ._llcrypt registered [11507.724104] Key type .llcrypt registered [11507.880185] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [11524.387845] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [11525.704754] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11525.863284] alg: No test for adler32 (adler32-zlib) [11526.992535] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [11527.287673] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [11528.975189] Key type lgssc registered [11530.330709] Lustre: Echo OBD driver; http://www.lustre.org/ [11564.705155] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 11:57:15 (1788451035) [11571.284881] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11571.322264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11572.652449] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11572.701673] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11572.821383] Lustre: lustre-MDT0000: new disk, initializing [11572.946725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11572.980915] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11578.094961] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [11589.255856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11589.375688] Lustre: 326731:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11589.431454] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11589.435659] Lustre: Skipped 1 previous similar message [11589.504978] Lustre: lustre-MDT0001: new disk, initializing [11589.599329] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11589.648709] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11589.664164] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11594.476902] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [11605.009879] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11611.956512] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11619.008703] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11619.262996] Lustre: lustre-OST0000: new disk, initializing [11619.266761] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11619.273165] Lustre: 328703:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11619.371175] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11626.099421] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [11627.631517] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11627.640940] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11627.768967] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11636.093044] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11653.600942] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11653.613832] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11653.635122] Lustre: Skipped 1 previous similar message [11653.636673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11653.660510] Lustre: Skipped 1 previous similar message [11657.117588] Lustre: server umount lustre-OST0000 complete [11661.282162] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11661.289449] LustreError: Skipped 1 previous similar message [11661.293137] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11661.308066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11663.844728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11663.844776] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11667.165689] Lustre: server umount lustre-MDT0000 complete [11670.948415] LustreError: 326724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451144 with bad export cookie 5415141514114950482 [11670.949386] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11670.955392] LustreError: 326724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11671.281907] Lustre: server umount lustre-MDT0001 complete [11682.450440] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [11684.976911] Key type lgssc unregistered [11685.315511] LNet: 330610:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11685.328735] LNetError: 330610:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11685.344940] LNet: Removed LNI 192.168.204.157@tcp [11686.115899] Key type .llcrypt unregistered [11686.118163] Key type ._llcrypt unregistered [11706.690203] Key type ._llcrypt registered [11706.692496] Key type .llcrypt registered [11706.829355] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [11708.093372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11708.131763] alg: No test for adler32 (adler32-zlib) [11709.221309] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [11709.721087] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [11711.407160] Key type lgssc registered [11712.577058] Lustre: Echo OBD driver; http://www.lustre.org/ [11720.469795] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11720.486312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11721.961329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11727.059866] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [11731.428518] LustreError: 331891:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11731.440131] LustreError: 331891:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11736.237993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11736.544923] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11741.608755] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [11749.963175] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11755.528756] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11762.745172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11763.137841] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11769.416821] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [11777.497719] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11790.346404] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11792.183454] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11796.976467] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11796.985793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11796.995828] Lustre: Skipped 2 previous similar messages [11797.007871] Lustre: Skipped 2 previous similar messages [11799.800942] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.57@tcp (stopping) [11802.082201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11802.093277] Lustre: Skipped 2 previous similar messages [11802.382385] Lustre: server umount lustre-MDT0000 complete [11804.901349] LustreError: 331888:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11806.492887] LustreError: 331871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451280 with bad export cookie 5089494287204339913 [11806.497343] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11806.506452] LustreError: 331871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11807.202519] LustreError: 333897:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11807.222814] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11807.265074] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11810.014492] LustreError: 331886:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11812.907293] Lustre: server umount lustre-MDT0001 complete [11817.478876] Lustre: server umount lustre-OST0000 complete [11827.413139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11832.565421] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [11835.753784] Lustre: 336083:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.157@tcp [11838.110811] Lustre: server umount MGS complete [11847.742866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11848.135904] LustreError: 336656:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11848.157888] LustreError: 336656:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11848.308928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11853.115803] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [11858.914255] LustreError: 336657:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11858.926763] LustreError: 336657:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11861.784494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11862.331549] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11866.878952] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [11876.663976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11877.094797] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11883.959258] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [11892.177157] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11893.691183] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11899.852216] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11901.888903] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11908.067202] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11908.070919] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11908.088606] LustreError: Skipped 1 previous similar message [11908.089846] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11908.089853] Lustre: Skipped 2 previous similar messages [11908.132500] Lustre: Skipped 1 previous similar message [11909.124645] Lustre: server umount lustre-OST0000 complete [11913.698856] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11913.705058] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11919.329223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11919.355368] Lustre: Skipped 5 previous similar messages [11919.694635] Lustre: server umount lustre-MDT0000 complete [11923.426416] LustreError: 336651:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11923.457926] LustreError: 336651:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11923.554576] LustreError: 336636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451397 with bad export cookie 5089494287204340753 [11923.559978] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11923.574076] LustreError: 336636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11923.826512] Lustre: server umount lustre-MDT0001 complete [11936.299958] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [11939.698700] Key type lgssc unregistered [11940.098431] LNet: 340094:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11940.108136] LNetError: 340094:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11940.130792] LNet: Removed LNI 192.168.204.157@tcp [11941.346257] Key type .llcrypt unregistered [11941.347996] Key type ._llcrypt unregistered [11959.720457] Key type ._llcrypt registered [11959.728324] Key type .llcrypt registered [11959.935078] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [11977.741420] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [11978.736838] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11978.827283] alg: No test for adler32 (adler32-zlib) [11979.886481] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [11980.073596] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [11981.705265] Key type lgssc registered [11983.204230] Lustre: Echo OBD driver; http://www.lustre.org/ [12020.925815] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12020.946304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12022.351800] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12022.383353] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12022.508517] Lustre: lustre-MDT0000: new disk, initializing [12022.616719] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12022.655183] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12027.288959] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [12038.824850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12038.946643] Lustre: 343252:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12039.011586] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12039.022621] Lustre: Skipped 1 previous similar message [12039.097202] Lustre: lustre-MDT0001: new disk, initializing [12039.176206] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12039.224513] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12039.238560] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12044.091578] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [12053.981864] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12060.687474] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12068.092464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12068.424378] Lustre: lustre-OST0000: new disk, initializing [12068.427390] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12068.431696] Lustre: 345217:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12068.520445] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12074.022624] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12074.038202] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12074.217487] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12075.203780] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [12083.513022] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12087.152423] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12087.465666] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12092.727382] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12093.089382] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12100.064071] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12100.068343] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12100.086673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12100.113767] Lustre: Skipped 1 previous similar message [12100.139332] Lustre: Skipped 1 previous similar message [12101.417962] Lustre: server umount lustre-OST0000 complete [12110.317141] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12110.327244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12110.338123] Lustre: Skipped 1 previous similar message [12115.114546] Lustre: server umount lustre-MDT0000 complete [12118.947818] LustreError: 343243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451592 with bad export cookie 96592017344325529 [12118.953343] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12118.962620] LustreError: 343243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12119.290897] Lustre: server umount lustre-MDT0001 complete [12128.177747] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 12:06:39 (1788451599) [12145.158893] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [12156.204899] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [12196.290745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12196.597723] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12196.654124] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12196.788288] Lustre: lustre-MDT0000: new disk, initializing [12196.909853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12196.941770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12202.332812] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [12214.841951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12214.994322] Lustre: 349979:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12215.063200] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12215.070361] Lustre: Skipped 1 previous similar message [12215.154621] Lustre: lustre-MDT0001: new disk, initializing [12215.349631] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12215.417553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12215.432701] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12221.093848] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [12232.230710] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12238.645251] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12246.533260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12246.953943] Lustre: lustre-OST0000: new disk, initializing [12246.957163] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12246.966050] Lustre: 351950:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12247.082200] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12252.786974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12252.795349] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12252.941955] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12254.286218] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [12264.956795] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12304.359514] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12304.364057] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12304.375846] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12304.375854] Lustre: Skipped 1 previous similar message [12309.479216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12309.494923] Lustre: Skipped 2 previous similar messages [12309.681388] Lustre: server umount lustre-OST0000 complete [12314.594125] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12314.606718] Lustre: Skipped 1 previous similar message [12314.613505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12314.618364] Lustre: Skipped 1 previous similar message [12316.445877] Lustre: server umount lustre-MDT0000 complete [12319.713131] LustreError: 349991:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12319.736339] LustreError: 349991:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12320.610762] LustreError: 351949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451794 with bad export cookie 96592017344326411 [12320.612431] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12320.626266] LustreError: 351949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12321.253466] Lustre: server umount lustre-MDT0001 complete [12333.000393] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [12336.178736] Key type lgssc unregistered [12336.534331] LNet: 354123:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12336.545380] LNetError: 354123:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12336.572943] LNet: Removed LNI 192.168.204.157@tcp [12337.493338] Key type .llcrypt unregistered [12337.495984] Key type ._llcrypt unregistered [12358.823161] Key type ._llcrypt registered [12358.827227] Key type .llcrypt registered [12358.965458] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [12359.759377] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12359.860257] alg: No test for adler32 (adler32-zlib) [12361.049701] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12361.302284] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [12362.975737] Key type lgssc registered [12364.162506] Lustre: Echo OBD driver; http://www.lustre.org/ [12372.342478] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12372.358359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12373.820069] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12378.541392] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [12383.203225] LustreError: 355392:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12383.220472] LustreError: 355392:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12387.474293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12387.967188] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12392.667908] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [12400.690440] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12406.320880] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12412.883841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12413.134821] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12419.253630] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [12427.531912] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12443.112597] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12443.119712] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12443.125717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12448.226105] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12448.235236] Lustre: Skipped 1 previous similar message [12449.090232] Lustre: server umount lustre-OST0000 complete [12453.350234] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12453.359961] Lustre: Skipped 1 previous similar message [12453.364072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12453.366966] Lustre: Skipped 2 previous similar messages [12458.469054] LustreError: 356120:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12458.474373] Lustre: server umount lustre-MDT0000 complete [12458.484793] LustreError: 356120:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12462.796089] LustreError: 355374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451936 with bad export cookie 5932253650859267970 [12462.828642] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12464.320915] Lustre: server umount lustre-MDT0001 complete [12480.453526] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [12484.474322] Key type lgssc unregistered [12484.913916] LNet: 360875:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12484.923385] LNetError: 360875:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12484.965206] LNet: Removed LNI 192.168.204.157@tcp [12486.319842] Key type .llcrypt unregistered [12486.328981] Key type ._llcrypt unregistered [12505.419703] Key type ._llcrypt registered [12505.427016] Key type .llcrypt registered [12505.588941] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [12506.972543] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12506.992683] alg: No test for adler32 (adler32-zlib) [12508.088677] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12508.327676] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [12510.080532] Key type lgssc registered [12511.600837] Lustre: Echo OBD driver; http://www.lustre.org/ [12523.202074] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12523.258369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12525.170633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12531.528245] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [12534.761543] LustreError: 364574:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12534.783812] LustreError: 364574:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12539.890393] LustreError: 364575:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12543.042904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12543.420287] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12548.387398] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [12555.307895] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12560.737387] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12567.167863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12567.535190] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12574.640547] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [12582.568798] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12594.152525] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12594.159928] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12594.172011] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12595.172206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12595.190560] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12595.198335] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12598.308188] Lustre: server umount lustre-OST0000 complete [12604.896191] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12604.899976] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12604.910625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12604.918640] Lustre: Skipped 1 previous similar message [12608.714639] Lustre: server umount lustre-MDT0000 complete [12610.538748] LustreError: 365882:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12610.558484] LustreError: 365882:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12613.878230] LustreError: 364500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452087 with bad export cookie 5430762716260594521 [12613.887097] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12613.908561] LustreError: 364500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12614.222873] Lustre: server umount lustre-MDT0001 complete [12627.675661] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [12631.432794] Key type lgssc unregistered [12631.968631] LNet: 371985:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12631.974954] LNetError: 371985:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12631.992231] LNet: Removed LNI 192.168.204.157@tcp [12632.917932] Key type .llcrypt unregistered [12632.919985] Key type ._llcrypt unregistered [12650.694258] Key type ._llcrypt registered [12650.704189] Key type .llcrypt registered [12650.897245] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [12652.156803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12652.195274] alg: No test for adler32 (adler32-zlib) [12653.332841] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12653.671484] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [12655.383154] Key type lgssc registered [12656.693746] Lustre: Echo OBD driver; http://www.lustre.org/ [12665.858398] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12665.904433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12667.683592] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12672.331705] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [12678.127421] LustreError: 375660:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12678.177040] LustreError: 375660:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12683.161381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12683.241152] LustreError: 375661:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12683.817973] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12689.642120] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [12698.012732] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12705.228362] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12713.496079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12714.166476] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12723.973265] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [12732.219701] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12754.401567] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12754.412632] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12754.416185] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12754.436874] Lustre: Skipped 1 previous similar message [12754.451181] Lustre: Skipped 1 previous similar message [12758.339251] Lustre: server umount lustre-OST0000 complete [12764.644532] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12764.649515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12764.667986] Lustre: Skipped 1 previous similar message [12764.686341] Lustre: Skipped 1 previous similar message [12768.723104] Lustre: server umount lustre-MDT0000 complete [12769.761256] LustreError: 375660:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12769.786614] LustreError: 375660:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12772.368669] LustreError: 375584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452246 with bad export cookie 9003142644794749750 [12772.374322] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12772.383637] LustreError: 375584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12772.832929] Lustre: server umount lustre-MDT0001 complete [12784.518192] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [12787.567628] Key type lgssc unregistered [12787.861970] LNet: 381735:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12787.874949] LNetError: 381735:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12787.891056] LNet: Removed LNI 192.168.204.157@tcp [12788.747183] Key type .llcrypt unregistered [12788.752531] Key type ._llcrypt unregistered [12890.845439] Key type ._llcrypt registered [12890.849596] Key type .llcrypt registered [12891.021155] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [12891.388663] Key type .llcrypt unregistered [12891.394989] Key type ._llcrypt unregistered [12967.883683] Key type ._llcrypt registered [12967.888121] Key type .llcrypt registered [12967.971854] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [12968.255463] Key type .llcrypt unregistered [12968.260568] Key type ._llcrypt unregistered [12999.233525] Key type ._llcrypt registered [12999.235540] Key type .llcrypt registered [12999.479617] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [12999.965816] Key type .llcrypt unregistered [12999.969046] Key type ._llcrypt unregistered [13033.616361] Key type ._llcrypt registered [13033.618597] Key type .llcrypt registered [13033.751756] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [13034.022705] Key type .llcrypt unregistered [13034.024393] Key type ._llcrypt unregistered [13115.762374] Key type ._llcrypt registered [13115.765264] Key type .llcrypt registered [13115.889357] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [13116.243332] Key type .llcrypt unregistered [13116.247394] Key type ._llcrypt unregistered [13149.849963] Key type ._llcrypt registered [13149.856683] Key type .llcrypt registered [13150.031027] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [13150.329338] Key type .llcrypt unregistered [13150.331633] Key type ._llcrypt unregistered [13185.052200] Key type ._llcrypt registered [13185.058505] Key type .llcrypt registered [13185.150478] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [13185.455319] Key type .llcrypt unregistered [13185.457122] Key type ._llcrypt unregistered [13221.121941] Key type ._llcrypt registered [13221.123902] Key type .llcrypt registered [13221.219849] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [13221.449418] Key type .llcrypt unregistered [13221.452593] Key type ._llcrypt unregistered [13257.170747] Key type ._llcrypt registered [13257.173406] Key type .llcrypt registered [13257.326261] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [13257.850473] Key type .llcrypt unregistered [13257.857353] Key type ._llcrypt unregistered [13292.162121] Key type ._llcrypt registered [13292.166970] Key type .llcrypt registered [13292.372412] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [13292.780654] Key type .llcrypt unregistered [13292.790939] Key type ._llcrypt unregistered [13363.004171] Key type ._llcrypt registered [13363.009887] Key type .llcrypt registered [13363.222475] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [13363.480613] Key type .llcrypt unregistered [13363.482366] Key type ._llcrypt unregistered [13393.060784] Key type ._llcrypt registered [13393.069502] Key type .llcrypt registered [13393.211474] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [13393.527296] Key type .llcrypt unregistered [13393.532294] Key type ._llcrypt unregistered [13429.426119] Key type ._llcrypt registered [13429.427868] Key type .llcrypt registered [13429.621345] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [13430.046049] Key type .llcrypt unregistered [13430.049148] Key type ._llcrypt unregistered [13483.118472] Key type ._llcrypt registered [13483.121300] Key type .llcrypt registered [13483.238542] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [13499.571477] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [13500.990823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13501.041894] alg: No test for adler32 (adler32-zlib) [13502.136437] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [13502.463648] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [13504.183165] Key type lgssc registered [13505.585704] Lustre: Echo OBD driver; http://www.lustre.org/ [13542.283803] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13542.317951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13543.598550] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13543.645667] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13543.763256] Lustre: tst250fs-MDT0000: new disk, initializing [13543.839600] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13543.859237] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13548.002635] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [13558.166266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13558.323814] Lustre: 402267:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13558.392502] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13558.399660] Lustre: Skipped 1 previous similar message [13558.487523] Lustre: tst250fs-MDT0001: new disk, initializing [13558.583447] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13558.647556] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13558.657133] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13563.227950] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [13568.168281] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13574.126283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13574.455213] Lustre: tst250fs-OST0000: Not available for connect from 0@lo (not set up) [13574.464723] Lustre: tst250fs-OST0000: new disk, initializing [13574.471934] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13574.484314] Lustre: 403901:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13574.599972] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13579.808182] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13579.823962] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13579.877568] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13581.120977] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [13598.729162] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13598.957532] Lustre: tst250fs-OST0001: new disk, initializing [13598.967664] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13598.981460] Lustre: 404772:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13599.205291] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13604.447803] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13604.460016] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13604.573730] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13608.479938] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [13614.816779] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13616.981757] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [13616.987783] Lustre: Skipped 1 previous similar message [13640.683490] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13640.692305] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13640.699873] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13640.714618] Lustre: Skipped 2 previous similar messages [13644.168812] Lustre: server umount tst250fs-MDT0000 complete [13648.126298] LustreError: 402260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453122 with bad export cookie 16117612604603000114 [13648.139736] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13648.597537] Lustre: server umount tst250fs-MDT0001 complete [13662.352183] Lustre: server umount tst250fs-OST0000 complete [13675.266361] Lustre: server umount tst250fs-OST0001 complete [13698.065143] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [13711.121687] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [13748.043618] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 12:33:39 (1788453219) [13750.269930] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13770.260687] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [13780.920697] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [13815.390636] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13486 sec ========= 12:34:46 (1788453286) [13817.223872] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:34:48 (1788453288) === [13821.209571] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:34:52 (1788453292) === [13865.200338] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [13868.555773] Key type lgssc unregistered [13868.908228] LNet: 413904:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13868.917124] LNetError: 413904:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13868.935057] LNet: Removed LNI 192.168.204.157@tcp [13869.834234] Key type .llcrypt unregistered [13869.837398] Key type ._llcrypt unregistered