[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 466967475 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002379] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004018] kvm-guest: setup PV IPIs [ 0.007313] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010132] LSM: Security Framework initializing [ 0.011076] Yama: becoming mindful. [ 0.013019] SELinux: Initializing. [ 0.014015] *** VALIDATE selinux *** [ 0.021764] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025687] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027116] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028116] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029124] *** VALIDATE tmpfs *** [ 0.031356] *** VALIDATE proc *** [ 0.032253] *** VALIDATE cgroup *** [ 0.033015] *** VALIDATE cgroup2 *** [ 0.034276] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035185] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037026] Spectre V2 : User space: Vulnerable [ 0.038008] Speculative Store Bypass: Vulnerable [ 0.040623] debug: unmapping init [mem 0xffffffffb1a59000-0xffffffffb1a60fff] [ 0.042275] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043727] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044032] ... version: 2 [ 0.045014] ... bit width: 48 [ 0.046016] ... generic registers: 4 [ 0.047016] ... value mask: 0000ffffffffffff [ 0.048018] ... max period: 00007fffffffffff [ 0.049018] ... fixed-purpose events: 3 [ 0.050017] ... event mask: 000000070000000f [ 0.051325] rcu: Hierarchical SRCU implementation. [ 0.053825] smp: Bringing up secondary CPUs ... [ 0.054733] x86: Booting SMP configuration: [ 0.055030] .... node #0, CPUs: #1 #2 #3 [ 0.058790] smp: Brought up 1 node, 4 CPUs [ 0.060011] smpboot: Max logical packages: 1 [ 0.061015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.142019] node 0 deferred pages initialised in 79ms [ 0.145638] devtmpfs: initialized [ 0.147269] x86/mm: Memory block size: 128MB [ 0.151544] gcov: version magic: 0x41383552 [ 0.155314] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.156178] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.157285] pinctrl core: initialized pinctrl subsystem [ 0.158146] [ 0.158560] ************************************************************* [ 0.159013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.160012] ** ** [ 0.161010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.162012] ** ** [ 0.163011] ** This means that this kernel is built to expose internal ** [ 0.164012] ** IOMMU data structures, which may compromise security on ** [ 0.165013] ** your system. ** [ 0.166014] ** ** [ 0.167013] ** If you see this message and you are not debugging the ** [ 0.168013] ** kernel, report this immediately to your vendor! ** [ 0.169014] ** ** [ 0.170009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.171012] ************************************************************* [ 0.172863] NET: Registered protocol family 16 [ 0.175583] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.177055] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.180058] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.184053] cpuidle: using governor menu [ 0.185886] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.189532] PCI: Using configuration type 1 for base access [ 0.191140] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.202126] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.205020] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.210071] cryptd: max_cpu_qlen set to 1000 [ 0.214353] ACPI: Added _OSI(Module Device) [ 0.216014] ACPI: Added _OSI(Processor Device) [ 0.217014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.219069] ACPI: Added _OSI(Processor Aggregator Device) [ 0.224206] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.231461] ACPI: Interpreter enabled [ 0.233099] ACPI: PM: (supports S0 S3 S4 S5) [ 0.234008] ACPI: Using IOAPIC for interrupt routing [ 0.236094] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.239631] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.250796] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.253054] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.256017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.260075] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.264439] acpiphp: Slot [2] registered [ 0.266059] acpiphp: Slot [5] registered [ 0.268102] acpiphp: Slot [6] registered [ 0.269102] acpiphp: Slot [7] registered [ 0.271112] acpiphp: Slot [8] registered [ 0.272098] acpiphp: Slot [9] registered [ 0.274104] acpiphp: Slot [10] registered [ 0.275102] acpiphp: Slot [3] registered [ 0.277093] acpiphp: Slot [4] registered [ 0.279123] acpiphp: Slot [11] registered [ 0.281189] acpiphp: Slot [12] registered [ 0.283105] acpiphp: Slot [13] registered [ 0.284103] acpiphp: Slot [14] registered [ 0.286104] acpiphp: Slot [15] registered [ 0.288148] acpiphp: Slot [16] registered [ 0.289101] acpiphp: Slot [17] registered [ 0.291107] acpiphp: Slot [18] registered [ 0.293122] acpiphp: Slot [19] registered [ 0.295108] acpiphp: Slot [20] registered [ 0.296142] acpiphp: Slot [21] registered [ 0.298105] acpiphp: Slot [22] registered [ 0.300103] acpiphp: Slot [23] registered [ 0.302105] acpiphp: Slot [24] registered [ 0.304099] acpiphp: Slot [25] registered [ 0.306111] acpiphp: Slot [26] registered [ 0.307085] acpiphp: Slot [27] registered [ 0.309224] acpiphp: Slot [28] registered [ 0.311084] acpiphp: Slot [29] registered [ 0.313109] acpiphp: Slot [30] registered [ 0.315232] acpiphp: Slot [31] registered [ 0.317067] PCI host bridge to bus 0000:00 [ 0.318016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.321113] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.323020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.326019] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.328019] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.331022] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.333179] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.335942] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.339142] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.348011] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.354036] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.357019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.359014] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.361014] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.364534] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.367913] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.371048] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.374940] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.381101] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.396015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.401013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.406550] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.421014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.429012] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.448016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.456913] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.463013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.469015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.487014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.498461] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.506023] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.515027] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.534025] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.548286] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.555019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.562015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.576022] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.589903] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.603027] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.617026] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.641019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.656546] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.664017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.674018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.694011] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.708741] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.712468] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.715458] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.718419] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.721301] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.726235] iommu: Default domain type: Passthrough [ 0.728468] SCSI subsystem initialized [ 0.729197] ACPI: bus type USB registered [ 0.731192] usbcore: registered new interface driver usbfs [ 0.734087] usbcore: registered new interface driver hub [ 0.736088] usbcore: registered new device driver usb [ 0.738175] pps_core: LinuxPPS API ver. 1 registered [ 0.740010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.742054] PTP clock support registered [ 0.743180] EDAC MC: Ver: 3.0.0 [ 0.745565] PCI: Using ACPI for IRQ routing [ 0.746551] NetLabel: Initializing [ 0.748016] NetLabel: domain hash size = 128 [ 0.749008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.751075] NetLabel: unlabeled traffic allowed by default [ 0.753266] vgaarb: loaded [ 0.755322] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.757013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.765166] clocksource: Switched to clocksource kvm-clock [ 0.882180] VFS: Disk quotas dquot_6.6.0 [ 0.884245] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.887063] *** VALIDATE ramfs *** [ 0.888845] *** VALIDATE hugetlbfs *** [ 0.890840] pnp: PnP ACPI init [ 0.894243] pnp: PnP ACPI: found 6 devices [ 0.912630] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.916300] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.918866] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.921739] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.923819] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.925729] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.928555] NET: Registered protocol family 2 [ 0.931483] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.936957] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.941248] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.946150] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.949225] TCP: Hash tables configured (established 65536 bind 65536) [ 0.951834] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.954864] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.957371] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.959820] NET: Registered protocol family 1 [ 0.962984] RPC: Registered named UNIX socket transport module. [ 0.965712] RPC: Registered udp transport module. [ 0.967459] RPC: Registered tcp transport module. [ 0.968986] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.971692] NET: Registered protocol family 44 [ 0.973784] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.976394] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.978649] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.981543] PCI: CLS 0 bytes, default 64 [ 0.983566] Unpacking initramfs... [ 2.393119] debug: unmapping init [mem 0xffff94d53cc54000-0xffff94d53ffbffff] [ 2.397335] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.398897] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.401740] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.911396] Initialise system trusted keyrings [ 2.912850] Key type blacklist registered [ 2.914539] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.928923] zbud: loaded [ 2.933546] *** VALIDATE nfs *** [ 2.934648] *** VALIDATE nfs4 *** [ 2.936669] pstore: using deflate compression [ 2.939832] Platform Keyring initialized [ 3.054098] NET: Registered protocol family 38 [ 3.056554] Key type asymmetric registered [ 3.058237] Asymmetric key parser 'x509' registered [ 3.060429] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.063877] io scheduler mq-deadline registered [ 3.065372] io scheduler kyber registered [ 3.066901] io scheduler bfq registered [ 3.068699] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.071674] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.074827] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.077950] ACPI: Power Button [PWRF] [ 3.082995] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.090264] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.112164] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.124192] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.147420] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.175486] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.206954] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.212834] Non-volatile memory driver v1.3 [ 3.214778] Linux agpgart interface v0.103 [ 3.249531] virtio_blk virtio1: [vda] 149768 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.253264] vda: detected capacity change from 0 to 76681216 [ 3.266489] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.268936] vdb: detected capacity change from 0 to 1073741824 [ 3.280972] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.283270] vdc: detected capacity change from 0 to 2621440000 [ 3.297253] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.300678] vdd: detected capacity change from 0 to 2621440000 [ 3.315474] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.318928] vde: detected capacity change from 0 to 4294967296 [ 3.334919] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.338660] vdf: detected capacity change from 0 to 4294967296 [ 3.355212] libphy: Fixed MDIO Bus: probed [ 3.360506] usbcore: registered new interface driver usbserial_generic [ 3.363332] usbserial: USB Serial support registered for generic [ 3.365801] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.369858] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.372166] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.374848] mousedev: PS/2 mouse device common for all mice [ 3.378693] rtc_cmos 00:05: RTC can wake from S4 [ 3.381871] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.386742] rtc_cmos 00:05: registered as rtc0 [ 3.388419] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.389894] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.395193] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.396474] intel_pstate: CPU model not supported [ 3.403743] hid: raw HID events driver (C) Jiri Kosina [ 3.406190] usbcore: registered new interface driver usbhid [ 3.408488] usbhid: USB HID core driver [ 3.410176] drop_monitor: Initializing network drop monitor service [ 3.412186] Initializing XFRM netlink socket [ 3.414426] NET: Registered protocol family 10 [ 3.417564] Segment Routing with IPv6 [ 3.418921] NET: Registered protocol family 17 [ 3.421039] mpls_gso: MPLS GSO support [ 3.427273] RAS: Correctable Errors collector initialized. [ 3.429831] AVX version of gcm_enc/dec engaged. [ 3.431888] AES CTR mode by8 optimization enabled [ 3.512959] sched_clock: Marking stable (3512928528, 0)->(4434637989, -921709461) [ 3.517500] registered taskstats version 1 [ 3.519978] Loading compiled-in X.509 certificates [ 3.522504] zswap: loaded using pool lzo/zbud [ 3.566166] Key type big_key registered [ 3.581728] Key type encrypted registered [ 3.583749] ima: No TPM chip found, activating TPM-bypass! [ 3.586361] ima: Allocated hash algorithm: sha1 [ 3.588443] ima: No architecture policies found [ 3.590125] evm: Initialising EVM extended attributes: [ 3.592088] evm: security.selinux [ 3.593546] evm: security.ima [ 3.594753] evm: security.capability [ 3.595976] evm: HMAC attrs: 0x1 [ 3.598449] rtc_cmos 00:05: setting system clock to 2026-09-05 16:06:00 UTC (1788624360) [ 3.607398] debug: unmapping init [mem 0xffffffffb2a03000-0xffffffffb2bfffff] [ 3.611264] debug: unmapping init [mem 0xffffffffb1782000-0xffffffffb1a58fff] [ 3.620094] Write protecting the kernel read-only data: 28672k [ 3.623693] debug: unmapping init [mem 0xffffffffafe03000-0xffffffffafffffff] [ 3.626986] debug: unmapping init [mem 0xffffffffb0714000-0xffffffffb07fffff] [ 3.669478] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.679891] systemd[1]: Detected virtualization kvm. [ 3.682507] systemd[1]: Detected architecture x86-64. [ 3.685283] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.714426] systemd[1]: No hostname configured. [ 3.716534] systemd[1]: Set hostname to . [ 3.719359] random: systemd: uninitialized urandom read (16 bytes read) [ 3.722448] systemd[1]: Initializing machine ID from random generator. [ 3.773602] random: ln: uninitialized urandom read (6 bytes read) [ 3.888449] random: systemd: uninitialized urandom read (16 bytes read) [ 3.891467] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.897600] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.905427] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.645524] device-mapper: uevent: version 1.0.3 [ 4.647965] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.444272] virtio_net virtio0 ens2: renamed from eth0 [ 5.459579] random: fast init done [ 5.499245] scsi host0: ata_piix [ 5.506654] scsi host1: ata_piix [ 5.508573] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.511295] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.262798] dracut-initqueue[580]: RTNETLINK answers: File exists [ 10.103686] random: crng init done [ 10.105296] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.650561] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.841176] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.114390] SELinux: Disabled at runtime. [ 12.179389] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.189854] systemd[1]: Detected virtualization kvm. [ 12.192052] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.695320] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.697949] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.703128] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.707651] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.710213] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.717352] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.725487] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. Mounting Huge Pages File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting POSIX Message Queue File System... [ OK ] Created slice system-serial\x2dgetty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Control Socket. [ 12.811971] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Mounting Kernel Debug File System... [ OK ] Created slice system-getty.slice. Starting udev Coldplug all Devices... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 13.180051] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.539955] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.541074] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.672171] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.694910] EDAC sbridge: Ver: 1.1.2 [ 15.377870] Key type dns_resolver registered [ 15.681425] NFS: Registering the id_resolver key type [ 15.683637] Key type id_resolver registered [ 15.685567] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started dnf makecache --timer. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg323-server login: [ 47.284122] libcfs: loading out-of-tree module taints kernel. [ 47.345458] Key type ._llcrypt registered [ 47.348454] Key type .llcrypt registered [ 47.432495] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 63.221261] hrtimer: interrupt took 10020899 ns [ 65.665408] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 66.988463] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 67.005227] alg: No test for adler32 (adler32-zlib) [ 68.474579] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [ 69.290542] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 71.080980] Key type lgssc registered [ 72.806230] Lustre: Echo OBD driver; http://www.lustre.org/ [ 87.197944] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 127.562971] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 139.510992] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 139.527703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 140.745162] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 140.777705] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 140.869764] Lustre: lustre-MDT0000: new disk, initializing [ 140.959303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 140.986175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 144.555591] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 156.202041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 156.307235] Lustre: 6501:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 156.351950] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 156.358809] Lustre: Skipped 1 previous similar message [ 156.473757] Lustre: lustre-MDT0001: new disk, initializing [ 156.542371] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 156.571358] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 156.583741] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 160.801308] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 165.030323] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 173.953071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 174.303971] Lustre: lustre-OST0000: new disk, initializing [ 174.311341] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 174.324208] Lustre: 8439:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 174.406624] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 179.216723] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 179.228321] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 179.318944] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 180.219766] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 194.458030] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 194.623852] Lustre: lustre-OST0001: new disk, initializing [ 194.632460] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 194.643203] Lustre: 9511:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 194.719950] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 201.166389] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 202.309983] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 202.320298] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 202.464827] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 212.830258] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 222.572848] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 229.099664] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing check_logdir /tmp/testlogs/ [ 234.163430] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing yml_node [ 238.433671] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 240.737486] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 243.135692] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 244.841853] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Sep 5 12:09:59 EDT 2026 [ 264.299596] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 266.263748] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 284.639873] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 284.649152] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 284.651061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 284.697716] Lustre: Skipped 3 previous similar messages [ 288.860912] Lustre: server umount lustre-MDT0000 complete [ 292.476560] LustreError: 6495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788624649 with bad export cookie 15994902371200670886 [ 292.479245] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 292.491102] LustreError: 6495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 294.883047] LustreError: 6508:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 294.883204] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 294.902426] LustreError: 6508:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 294.932549] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 294.942682] Lustre: Skipped 4 previous similar messages [ 297.892363] LustreError: 6508:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 297.892695] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 297.912378] LustreError: 6508:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 297.921444] Lustre: Skipped 1 previous similar message [ 298.886691] Lustre: server umount lustre-MDT0001 complete [ 303.194496] Lustre: server umount lustre-OST0000 complete [ 307.237284] Lustre: server umount lustre-OST0001 complete [ 313.302837] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 324.051952] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 361.821995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 362.102210] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 362.152552] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 362.264449] Lustre: lustre-MDT0000: new disk, initializing [ 362.360418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 362.377548] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 368.022332] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 378.983028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 379.117697] Lustre: 15066:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 379.190483] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 379.195449] Lustre: Skipped 1 previous similar message [ 379.303366] Lustre: lustre-MDT0001: new disk, initializing [ 379.432527] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 379.501649] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 379.512948] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 384.337247] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 394.735159] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 401.057487] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 407.811042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 408.033650] Lustre: lustre-OST0000: new disk, initializing [ 408.038426] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 408.048092] Lustre: 17036:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 408.167525] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 409.888352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 409.901065] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 410.055528] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 414.405336] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 424.660586] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 429.764524] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 429.980551] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 434.414712] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 434.679444] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 440.815764] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 440.820531] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 440.829617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 440.832257] LustreError: Skipped 1 previous similar message [ 440.864778] Lustre: Skipped 2 previous similar messages [ 442.665108] Lustre: server umount lustre-OST0000 complete [ 451.042618] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 451.043514] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 451.045597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 451.045606] Lustre: Skipped 1 previous similar message [ 451.070704] Lustre: Skipped 1 previous similar message [ 452.551968] Lustre: server umount lustre-MDT0000 complete [ 456.162918] LustreError: 17761:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 456.180689] LustreError: 17761:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 456.444845] LustreError: 15058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788624813 with bad export cookie 15994902371200672419 [ 456.450599] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 456.463023] LustreError: 15058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 456.781099] Lustre: server umount lustre-MDT0001 complete [ 462.640625] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 12:13:37 (1788624817) [ 481.055639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 481.340380] LustreError: 19910:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 481.388892] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 485.689623] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 486.882068] LustreError: 19911:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 496.009503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 502.507679] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 513.235392] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 519.698955] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 521.784599] Lustre: Failing over lustre-MDT0000 [ 522.057674] Lustre: server umount lustre-MDT0000 complete [ 522.212083] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 522.218236] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 522.234321] LustreError: 19911:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 522.256662] LustreError: 19911:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 541.669928] LustreError: 19907:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 541.715370] LustreError: 19907:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 542.351169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 542.527662] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 548.326848] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 548.334483] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 548.405581] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 548.775265] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 559.419946] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 563.856529] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 563.887546] Lustre: Skipped 1 previous similar message [ 563.898508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 563.910501] Lustre: Skipped 1 previous similar message [ 570.240956] Lustre: server umount lustre-MDT0000 complete [ 574.895421] LustreError: 19892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788624931 with bad export cookie 15994902371200673833 [ 574.913183] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 574.915198] LustreError: 19892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 575.632802] Lustre: server umount lustre-MDT0001 complete [ 586.052729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 586.552138] LustreError: 23248:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 586.565695] LustreError: 23248:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 586.613171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 586.620289] Lustre: Skipped 2 previous similar messages [ 591.379462] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 600.253728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 605.050622] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 612.556611] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 618.958008] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 621.181366] Lustre: Failing over lustre-MDT0000 [ 621.390277] Lustre: server umount lustre-MDT0000 complete [ 622.049436] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 622.065088] Lustre: Skipped 2 previous similar messages [ 637.343632] Lustre: 3637:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788624978/real 1788624978] req@ffff94d481cee300 x1875508646306048/t0(0) o400->MGC192.168.203.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788624994 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 637.369639] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 640.237708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 647.648865] LustreError: 3634:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff94d5bc3f2a00 x1875508646308992/t0(0) o250->MGC192.168.203.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 647.882774] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 651.958889] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 653.288691] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 653.317705] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 653.331345] Lustre: Skipped 1 previous similar message [ 653.349493] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 660.459224] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 667.624899] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 667.639573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 667.643659] Lustre: Skipped 1 previous similar message [ 667.662892] Lustre: Skipped 3 previous similar messages [ 671.040124] Lustre: server umount lustre-MDT0000 complete [ 674.912163] LustreError: 23230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788625031 with bad export cookie 15994902371200674771 [ 674.917567] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 674.927566] LustreError: 23230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 675.198728] Lustre: server umount lustre-MDT0001 complete [ 683.614608] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 12:17:18 (1788625038) [ 699.012271] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 709.900235] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 750.762104] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 760.711617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 761.065028] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 761.106568] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 761.196277] Lustre: lustre-MDT0000: new disk, initializing [ 761.266242] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 761.274412] Lustre: Skipped 2 previous similar messages [ 761.303617] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 765.632857] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 776.385396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 776.505806] Lustre: 29622:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 776.770049] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 781.614385] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 791.290479] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 795.875185] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 802.486033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 802.763886] Lustre: 31588:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 804.511862] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 804.579744] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 804.624045] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 804.648309] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 810.672349] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 821.736206] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 828.218979] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 856.034922] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 856.039930] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 856.050588] LustreError: Skipped 1 previous similar message [ 856.053587] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 856.053598] Lustre: Skipped 2 previous similar messages [ 856.123571] Lustre: Skipped 1 previous similar message [ 868.320240] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 868.555557] Lustre: server umount lustre-OST0000 complete [ 873.952148] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 878.944237] Lustre: server umount lustre-MDT0000 complete [ 881.632123] LustreError: 29630:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 881.659186] LustreError: 29630:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 19 previous similar messages [ 883.343433] LustreError: 32817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788625240 with bad export cookie 15994902371200675177 [ 883.359971] LustreError: 32817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 883.366260] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 884.127985] Lustre: server umount lustre-MDT0001 complete [ 898.194797] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 902.633770] Key type lgssc unregistered [ 903.243631] LNet: 33576:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 903.255424] LNetError: 33576:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 903.288959] LNet: Removed LNI 192.168.203.123@tcp [ 905.171749] Key type .llcrypt unregistered [ 905.174184] Key type ._llcrypt unregistered [ 930.395322] Key type ._llcrypt registered [ 930.397828] Key type .llcrypt registered [ 930.499819] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 931.511035] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 931.678871] alg: No test for adler32 (adler32-zlib) [ 932.878159] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [ 933.226734] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 934.967714] Key type lgssc registered [ 936.200835] Lustre: Echo OBD driver; http://www.lustre.org/ [ 946.176850] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 946.192671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 947.561419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 952.407628] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 957.923773] LustreError: 34989:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 957.948391] LustreError: 34989:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 961.968806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 962.436602] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 967.318117] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 975.404751] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 981.287982] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 987.899035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 988.202671] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 992.321966] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 994.264261] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1001.540951] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1007.674656] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1017.729504] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 12:22:52 (1788625372) [ 1019.498994] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1021.061486] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1027.366942] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 12:23:02 (1788625382) [ 1034.698707] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 12:23:09 (1788625389) [ 1035.762233] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1044.807574] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1044.814915] Lustre: Skipped 2 previous similar messages [ 1046.357559] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 12:23:21 (1788625401) [ 1050.351700] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1050.356739] Lustre: Skipped 3 previous similar messages [ 1068.636870] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1068.640695] Lustre: Skipped 6 previous similar messages [ 1071.228196] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 12:23:46 (1788625426) [ 1074.012982] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1077.813654] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1086.812156] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 12:24:01 (1788625441) [ 1089.008508] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1089.016201] Lustre: Skipped 3 previous similar messages [ 1102.648888] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 12:24:17 (1788625457) [ 1120.228853] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1120.228880] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1120.261873] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1120.270677] Lustre: Skipped 1 previous similar message [ 1122.580793] Lustre: server umount lustre-OST0000 complete [ 1126.372979] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1126.382911] LustreError: Skipped 1 previous similar message [ 1126.386382] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1126.395577] Lustre: Skipped 1 previous similar message [ 1126.400795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1130.465378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1130.465621] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1132.248654] Lustre: server umount lustre-MDT0000 complete [ 1135.586304] LustreError: 34989:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1135.626847] LustreError: 34989:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1136.118735] LustreError: 34969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788625493 with bad export cookie 4693610029691565001 [ 1136.119864] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1136.135678] LustreError: 34969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1136.502260] Lustre: server umount lustre-MDT0001 complete [ 1146.388532] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 1149.042728] Key type lgssc unregistered [ 1149.322801] LNet: 42846:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1149.328633] LNetError: 42846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1149.344066] LNet: Removed LNI 192.168.203.123@tcp [ 1150.109886] Key type .llcrypt unregistered [ 1150.113423] Key type ._llcrypt unregistered [ 1169.137712] Key type ._llcrypt registered [ 1169.141040] Key type .llcrypt registered [ 1169.263117] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 1186.559610] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 1187.435648] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1187.527161] alg: No test for adler32 (adler32-zlib) [ 1188.519359] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [ 1188.716849] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 1190.407176] Key type lgssc registered [ 1191.661271] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1230.764092] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1230.852193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1232.158160] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1232.213198] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1232.315304] Lustre: lustre-MDT0000: new disk, initializing [ 1232.425820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1232.466988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1237.524948] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1248.880776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1249.000529] Lustre: 45997:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1249.048417] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1249.053146] Lustre: Skipped 1 previous similar message [ 1249.146061] Lustre: lustre-MDT0001: new disk, initializing [ 1249.250636] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1249.305826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1249.320157] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1254.001434] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1263.770471] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1269.829776] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1277.133138] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1277.606841] Lustre: lustre-OST0000: new disk, initializing [ 1277.611883] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1277.620632] Lustre: 47963:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1277.769556] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1283.143801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1283.153927] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1283.317420] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1284.151918] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1294.473425] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1299.136205] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1299.337340] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1304.050303] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1304.352725] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1309.153836] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1309.154855] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1309.163170] LustreError: Skipped 1 previous similar message [ 1309.166679] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1309.188927] Lustre: Skipped 1 previous similar message [ 1309.210489] Lustre: Skipped 1 previous similar message [ 1312.700019] Lustre: server umount lustre-OST0000 complete [ 1319.394722] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1319.406266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1319.409483] Lustre: Skipped 1 previous similar message [ 1319.427992] Lustre: Skipped 1 previous similar message [ 1324.514047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1324.519589] Lustre: Skipped 1 previous similar message [ 1329.633910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1329.643070] Lustre: Skipped 1 previous similar message [ 1331.168128] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1333.354699] Lustre: server umount lustre-MDT0000 complete [ 1336.942400] LustreError: 45988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788625693 with bad export cookie 1596597137277048537 [ 1336.951503] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1336.956036] LustreError: 45988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1337.460024] Lustre: server umount lustre-MDT0001 complete [ 1348.087357] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 12:28:22 (1788625702) [ 1355.386490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1355.860840] LustreError: 50427:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1355.877411] LustreError: 50427:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1355.927811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1359.997648] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1361.380577] LustreError: 50428:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1365.475873] LustreError: 50427:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1367.826809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1368.082384] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1372.616197] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1381.329375] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1389.757757] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1399.210663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1399.512242] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1406.457983] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1414.475368] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1419.622129] LustreError: 52956:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1420.832665] Lustre: Modifying parameter general.timeout=1 in log params [ 1421.973765] Lustre: Modifying parameter general.timeout=2 in log params [ 1424.172893] Lustre: Modifying parameter general.timeout=4 in log params [ 1424.181499] Lustre: Skipped 1 previous similar message [ 1428.715587] Lustre: Modifying parameter general.timeout=9 in log params [ 1428.729901] Lustre: Skipped 4 previous similar messages [ 1437.408485] Lustre: Modifying parameter general.timeout=20 in log params [ 1437.414158] Lustre: Skipped 10 previous similar messages [ 1454.107737] Lustre: Modifying parameter general.timeout=39 in log params [ 1454.115696] Lustre: Skipped 18 previous similar messages [ 1486.833719] Lustre: Modifying parameter general.timeout=76 in log params [ 1486.849506] Lustre: Skipped 36 previous similar messages [ 1549.003192] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 12:31:43 (1788625903) [ 1551.651258] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 in log params [ 1551.665898] Lustre: Skipped 55 previous similar messages [ 1552.660943] Lustre: 59897:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1554.776299] Lustre: 60006:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1556.904684] Lustre: 60105:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1562.677492] Lustre: 60419:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1562.685218] Lustre: 60419:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1591.434254] Lustre: 61779:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1591.443439] LustreError: 61779:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1596.014300] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 12:32:30 (1788625950) [ 1608.674552] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1608.678488] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1608.701497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1612.258430] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1612.275238] Lustre: Skipped 1 previous similar message [ 1614.620820] Lustre: server umount lustre-MDT0000 complete [ 1617.377682] LustreError: 50423:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1617.396708] LustreError: 50423:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1618.079095] LustreError: 50408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788625974 with bad export cookie 1596597137277049412 [ 1618.083124] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1618.093423] LustreError: 50408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1618.474227] Lustre: server umount lustre-MDT0001 complete [ 1632.829284] Lustre: server umount lustre-OST0000 complete [ 1661.442620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1661.780209] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1662.210304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1667.514962] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1677.830837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1677.922219] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1677.941471] Lustre: 64310:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1683.094868] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1693.431583] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1699.403579] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1707.134442] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1707.464497] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1707.912445] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1707.917707] Lustre: Skipped 1 previous similar message [ 1717.459768] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1727.669567] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1729.535111] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1729.550479] Lustre: Skipped 21 previous similar messages [ 1729.623124] Lustre: 66878:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1737.696739] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1737.697952] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1737.703030] LustreError: Skipped 1 previous similar message [ 1737.707373] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1737.713429] Lustre: Skipped 1 previous similar message [ 1737.733267] Lustre: Skipped 4 previous similar messages [ 1742.840252] Lustre: server umount lustre-OST0000 complete [ 1747.944654] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1752.635513] Lustre: server umount lustre-MDT0000 complete [ 1753.058632] LustreError: 65742:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1753.087358] LustreError: 65742:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1757.021863] LustreError: 64301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788626113 with bad export cookie 1596597137277052464 [ 1757.042104] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1757.428829] Lustre: server umount lustre-MDT0001 complete [ 1770.898745] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 1773.552831] Key type lgssc unregistered [ 1774.008689] LNet: 68085:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1774.023532] LNetError: 68085:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1774.062189] LNet: Removed LNI 192.168.203.123@tcp [ 1775.144987] Key type .llcrypt unregistered [ 1775.146904] Key type ._llcrypt unregistered [ 1804.252831] Key type ._llcrypt registered [ 1804.256609] Key type .llcrypt registered [ 1804.392883] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 1805.577726] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1805.736161] alg: No test for adler32 (adler32-zlib) [ 1806.946670] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [ 1807.282703] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 1808.935407] Key type lgssc registered [ 1810.426158] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1818.988199] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1819.011279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1820.637815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1825.627206] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1830.886399] LustreError: 69497:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1830.905939] LustreError: 69497:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1831.919903] LustreError: 69496:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1835.044473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1835.869682] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1840.859044] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1849.398991] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1855.630640] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1862.651109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1863.378660] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1870.296854] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1877.874589] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1888.621246] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1901.538017] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1901.546029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1901.552590] Lustre: Skipped 1 previous similar message [ 1901.575968] Lustre: Skipped 2 previous similar messages [ 1904.406804] Lustre: server umount lustre-MDT0000 complete [ 1906.678320] LustreError: 69492:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1906.698332] LustreError: 69492:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1908.526493] LustreError: 69478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788626265 with bad export cookie 3997965710495015379 [ 1908.550541] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1908.906910] Lustre: server umount lustre-MDT0001 complete [ 1922.382960] Lustre: server umount lustre-OST0000 complete [ 1954.029227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1954.416621] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1954.471511] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1954.932645] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1961.152706] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1970.711841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1970.810953] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1970.831740] Lustre: 74366:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1971.522874] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1977.108343] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1986.103065] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1993.292560] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1999.980080] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2000.229070] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2000.620101] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2008.004938] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2016.973704] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2019.154574] Lustre: 76943:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2021.406451] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2027.493952] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2027.501788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2027.505852] Lustre: Skipped 2 previous similar messages [ 2027.506595] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2027.538124] Lustre: Skipped 1 previous similar message [ 2032.612917] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2032.622620] Lustre: Skipped 1 previous similar message [ 2033.846464] Lustre: server umount lustre-OST0000 complete [ 2042.848775] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2042.855459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2042.860142] Lustre: Skipped 1 previous similar message [ 2042.867767] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2042.871337] Lustre: Skipped 1 previous similar message [ 2044.772493] Lustre: server umount lustre-MDT0000 complete [ 2047.969790] LustreError: 74377:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2047.995542] LustreError: 74377:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2049.574697] LustreError: 74357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788626406 with bad export cookie 3997965710495016226 [ 2049.576157] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2049.598537] LustreError: 74357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 2050.008085] Lustre: server umount lustre-MDT0001 complete [ 2063.404248] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 2067.000411] Key type lgssc unregistered [ 2067.522293] LNet: 78154:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2067.531960] LNetError: 78154:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2067.554545] LNet: Removed LNI 192.168.203.123@tcp [ 2068.818964] Key type .llcrypt unregistered [ 2068.827274] Key type ._llcrypt unregistered [ 2097.710236] Key type ._llcrypt registered [ 2097.713243] Key type .llcrypt registered [ 2097.873552] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 2099.162628] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2099.397076] alg: No test for adler32 (adler32-zlib) [ 2100.567669] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [ 2100.953126] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 2102.719536] Key type lgssc registered [ 2104.253079] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2113.300901] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2113.355282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2115.054893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2120.455859] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2124.283623] LustreError: 79567:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2124.305341] LustreError: 79567:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2129.378811] LustreError: 79568:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2130.731898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2131.447898] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2136.563912] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2144.670803] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2151.559930] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2159.525746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2160.145893] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2166.560358] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2175.563647] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2192.614599] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2193.633366] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2195.321259] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2195.327970] Lustre: Skipped 1 previous similar message [ 2197.346220] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2197.352670] Lustre: Skipped 1 previous similar message [ 2201.487663] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2201.491065] Lustre: Skipped 3 previous similar messages [ 2210.417193] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2210.429190] Lustre: Skipped 7 previous similar messages [ 2226.668177] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2226.673504] Lustre: Skipped 15 previous similar messages [ 2258.669366] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2258.677928] Lustre: Skipped 29 previous similar messages [ 2305.428689] Lustre: 87693:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2308.744448] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 12:44:23 (1788626663) [ 2310.801902] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2313.372887] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 12:44:27 (1788626667) [ 2320.873037] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2320.880297] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2320.894358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2322.912541] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2322.917228] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2322.939206] Lustre: Skipped 1 previous similar message [ 2324.572938] Lustre: server umount lustre-OST0000 complete [ 2328.032601] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2328.047925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2328.056580] Lustre: Skipped 1 previous similar message [ 2333.152646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2333.160085] Lustre: Skipped 1 previous similar message [ 2338.279896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2338.291501] Lustre: Skipped 1 previous similar message [ 2342.367356] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2342.521137] Lustre: server umount lustre-MDT0000 complete [ 2343.406707] LustreError: 79562:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2343.433233] LustreError: 79562:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2346.279293] LustreError: 79547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788626703 with bad export cookie 9463432411298137683 [ 2346.285388] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2346.294465] LustreError: 79547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2346.567693] Lustre: server umount lustre-MDT0001 complete [ 2359.049959] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 2362.335560] Key type lgssc unregistered [ 2362.728295] LNet: 88943:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2362.765110] LNetError: 88943:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2363.814089] LNet: Removed LNI 192.168.203.123@tcp [ 2364.714916] Key type .llcrypt unregistered [ 2364.717198] Key type ._llcrypt unregistered [ 2369.404846] Key type ._llcrypt registered [ 2369.407281] Key type .llcrypt registered [ 2369.519888] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2374.412734] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules [ 2375.342094] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2375.422219] alg: No test for adler32 (adler32-zlib) [ 2376.408352] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [ 2376.607350] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 2378.288175] Key type lgssc registered [ 2379.460379] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2380.694126] LustreError: 90141:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2398.452155] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 2402.686737] LustreError: 90141:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2407.228931] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2407.251257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2408.842302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2414.074211] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2418.145047] LustreError: 91168:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2418.177910] LustreError: 91168:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2423.264674] LustreError: 91169:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2424.876480] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 12:46:19 (1788626779) [ 2428.414761] LustreError: 91168:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2431.868855] LustreError: 91201:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2431.878636] LustreError: 91201:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2431.884244] LustreError: 91201:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 23, retries 0, failed: rc = -5 [ 2434.348279] Lustre: server umount lustre-MDT0000 complete [ 2454.521143] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 2457.306967] Key type lgssc unregistered [ 2457.584908] LNet: 92696:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2457.594110] LNetError: 92696:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2457.619387] LNet: Removed LNI 192.168.203.123@tcp [ 2458.536357] Key type .llcrypt unregistered [ 2458.539543] Key type ._llcrypt unregistered [ 2475.339334] Key type ._llcrypt registered [ 2475.342100] Key type .llcrypt registered [ 2475.438832] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 2476.844306] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2476.878458] alg: No test for adler32 (adler32-zlib) [ 2477.954047] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [ 2478.229675] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 2479.887658] Key type lgssc registered [ 2481.082898] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2488.993574] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2489.024428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2490.478878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2495.346885] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2501.097680] LustreError: 93909:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2501.122745] LustreError: 93909:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2503.860813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2504.487328] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2509.487953] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2518.163451] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2525.413950] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2534.072280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2535.456444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2543.980832] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2553.421216] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2607.918842] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 12:49:22 (1788626962) [ 2609.828666] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2611.994594] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 12:49:26 (1788626966) [ 2616.801662] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2616.808784] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2616.827157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2621.416490] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2621.419494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2621.441518] Lustre: Skipped 1 previous similar message [ 2621.468657] Lustre: Skipped 2 previous similar messages [ 2622.840987] Lustre: server umount lustre-MDT0000 complete [ 2626.530035] LustreError: 93910:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2626.571320] LustreError: 93910:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2627.587840] LustreError: 93890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788626984 with bad export cookie 9284191056284810350 [ 2627.626255] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2628.030961] Lustre: server umount lustre-MDT0001 complete [ 2641.701931] Lustre: server umount lustre-OST0000 complete [ 2655.529354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2656.053827] LustreError: 98205:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2656.305698] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2661.344685] LustreError: 98206:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2661.927687] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2665.447529] LustreError: 98205:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2671.875908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2672.810875] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2678.099303] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2686.057379] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2691.788797] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2706.646729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2706.856128] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2706.863103] LustreError: 98194:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2706.890826] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2706.913802] LustreError: 100288:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -98 [ 2706.919884] LustreError: 100288:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2706.926071] LustreError: 100288:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2707.020385] Lustre: server umount lustre-OST0000 complete [ 2707.022727] LustreError: 100288:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2715.829941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2716.067811] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2716.077145] LustreError: 98194:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2716.124643] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2716.145484] LustreError: 100754:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -98 [ 2716.157628] LustreError: 100754:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2716.166918] LustreError: 100754:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2716.239046] Lustre: server umount lustre-OST0000 complete [ 2716.241245] LustreError: 100754:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2727.190893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2727.412678] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2727.422039] Lustre: Found index 0 for lustre-OST0000, updating log [ 2727.430474] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2727.509574] Lustre: lustre-OST0000: new disk, initializing [ 2727.513760] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2727.521329] Lustre: 101292:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2727.843901] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2734.874499] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2739.154922] Lustre: Failing over lustre-OST0000 [ 2739.395941] Lustre: server umount lustre-OST0000 complete [ 2747.365487] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2747.368909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2747.392203] Lustre: Skipped 1 previous similar message [ 2750.651512] Lustre: server umount lustre-MDT0000 complete [ 2752.507152] LustreError: 98206:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2752.530202] LustreError: 98206:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2755.216136] LustreError: 98185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788627112 with bad export cookie 9284191056284813101 [ 2755.220156] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2755.239972] LustreError: 98185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 2755.711764] Lustre: server umount lustre-MDT0001 complete [ 2765.581727] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 12:52:00 (1788627120) [ 2786.253872] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 2799.226934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2799.881798] LustreError: 103637:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2800.006517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2805.704480] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2815.339453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2821.131833] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2824.746237] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2833.067438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2833.687812] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2833.696989] Lustre: Skipped 1 previous similar message [ 2842.063190] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2852.967095] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2853.060516] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2853.154813] Lustre: lustre-OST0001: new disk, initializing [ 2853.158806] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2853.167454] Lustre: 105957:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2860.611571] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2862.182393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2862.202933] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2862.406136] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2870.153900] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2879.097457] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2884.469474] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.23@tcp (stopping) [ 2884.475212] Lustre: Skipped 1 previous similar message [ 2887.655723] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2887.663628] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2889.557224] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.23@tcp (stopping) [ 2889.568405] Lustre: Skipped 4 previous similar messages [ 2898.399797] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2898.407301] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2898.413715] Lustre: Skipped 6 previous similar messages [ 2898.675253] Lustre: server umount lustre-MDT0001 complete [ 2899.794015] LustreError: 103632:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2899.822182] LustreError: 103632:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2910.537714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2910.621555] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2910.625264] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2910.680181] Lustre: 103627:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2911.874651] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2911.892892] Lustre: Skipped 1 previous similar message [ 2912.457477] LustreError: 107492:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2912.466225] LustreError: 107492:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2912.488621] LustreError: 103630:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2912.588207] LustreError: 107500:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.123@tcp: cfg command failed: rc = -17 [ 2912.594071] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2912.594071] [ 2917.390133] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 2917.748804] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2930.382653] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 12:54:44 (1788627284) [ 2950.090813] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 2972.852802] Lustre: 109573:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2998.736211] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3003.190737] Lustre: 110708:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3095.526887] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3095.536375] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3095.559269] Lustre: Skipped 2 previous similar messages [ 3095.570448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3098.319839] Lustre: server umount lustre-MDT0000 complete [ 3101.664679] LustreError: 103632:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3101.692825] LustreError: 103632:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 3102.039075] LustreError: 105155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788627458 with bad export cookie 9284191056284814095 [ 3102.045166] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3102.057708] LustreError: 105155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3102.439495] Lustre: server umount lustre-MDT0001 complete [ 3117.166327] Lustre: server umount lustre-OST0000 complete [ 3130.600554] Lustre: server umount lustre-OST0001 complete [ 3136.953684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3149.205405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3158.382933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3170.690333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3192.595242] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 3204.759829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3204.775906] Lustre: lustre-MDT0000: reset Object Index mappings [ 3205.422179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3210.427662] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3219.656439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3219.677666] Lustre: lustre-MDT0001: reset Object Index mappings [ 3225.090251] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3228.550923] Lustre: 115810:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3236.957514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3244.969330] LustreError: 116164:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3244.990392] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 3245.008317] LustreError: 116164:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3245.262457] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3251.197837] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:349 to 0x280000401:386) [ 3254.488894] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3260.914214] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:157 to 0x2c0000401:193) [ 3260.917916] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:199 to 0x2c0000400:225) [ 3262.417870] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3271.213353] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3275.089386] Lustre: 117687:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3329.859762] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 13:01:24 (1788627684) [ 3358.183140] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3358.203065] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3358.203230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3358.203235] Lustre: Skipped 4 previous similar messages [ 3358.212612] Lustre: Skipped 4 previous similar messages [ 3361.752598] Lustre: server umount lustre-MDT0000 complete [ 3365.621687] LustreError: 114644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788627722 with bad export cookie 9284191056284860967 [ 3365.637227] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3365.637718] LustreError: 114644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3372.131230] Lustre: server umount lustre-MDT0001 complete [ 3382.039948] Lustre: server umount lustre-OST0000 complete [ 3391.899452] Lustre: server umount lustre-OST0001 complete [ 3399.337303] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 3409.916386] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 3452.063208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3452.421276] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3452.498616] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3452.683939] Lustre: lustre-MDT0000: new disk, initializing [ 3452.843029] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3458.275604] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3466.070603] Lustre: server umount lustre-MDT0000 complete [ 3477.956823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3478.478549] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3478.512629] Lustre: 121975:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3478.527759] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3478.868405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3478.873281] Lustre: Skipped 4 previous similar messages [ 3483.348358] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3488.275604] Lustre: server umount lustre-MDT0000 complete [ 3500.090692] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 13:04:14 (1788627854) [ 3501.802692] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3504.211218] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 13:04:18 (1788627858) [ 3526.712113] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 3538.813819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3543.988252] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3553.028659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3553.186737] Lustre: 124048:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3553.283178] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3553.296243] Lustre: Skipped 1 previous similar message [ 3553.446868] Lustre: lustre-MDT0001: new disk, initializing [ 3553.648358] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3553.660049] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3558.290247] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3563.124686] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3569.940326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3570.198867] Lustre: lustre-OST0000: new disk, initializing [ 3570.202991] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3570.209239] Lustre: 125625:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3571.909239] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3571.923455] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3572.061501] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3576.767747] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3588.486733] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3588.701506] Lustre: lustre-OST0001: new disk, initializing [ 3588.706624] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3588.718542] Lustre: 126494:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3590.598376] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3590.609491] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3590.694103] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3596.309068] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3606.400842] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3610.229405] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4004.555590] Lustre: 132921:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4004.562575] Lustre: 132921:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4004.567989] Lustre: 132921:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4004.580732] Lustre: 132921:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4004.588856] Lustre: 132921:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4004.598808] Lustre: 132921:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4310.030614] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 13:17:43 (1788628663) [ 4326.882362] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4326.900388] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4326.931943] Lustre: Skipped 4 previous similar messages [ 4326.937845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4326.942093] Lustre: Skipped 7 previous similar messages [ 4338.145670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4338.172941] Lustre: Skipped 11 previous similar messages [ 4340.194707] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4340.547691] Lustre: server umount lustre-MDT0000 complete [ 4343.267679] LustreError: 124875:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4343.278594] LustreError: 124875:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 4344.321971] LustreError: 126493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788628701 with bad export cookie 9284191056284929868 [ 4344.324282] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4344.344017] LustreError: 126493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4348.392876] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4348.403578] Lustre: Skipped 4 previous similar messages [ 4351.018917] Lustre: server umount lustre-MDT0001 complete [ 4369.375158] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4369.520768] Lustre: server umount lustre-OST0000 complete [ 4379.470249] Lustre: server umount lustre-OST0001 complete [ 4385.978300] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 4396.646694] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 4439.746659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4439.979805] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4440.002250] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4440.063422] Lustre: lustre-MDT0000: new disk, initializing [ 4440.126024] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4440.134300] Lustre: Skipped 4 previous similar messages [ 4440.147173] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4445.066970] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4455.581534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4455.681976] Lustre: 135981:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4455.743613] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4455.748060] Lustre: Skipped 1 previous similar message [ 4455.849105] Lustre: lustre-MDT0001: new disk, initializing [ 4455.984157] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4455.994244] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4460.665050] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4471.461502] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4479.608957] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4489.070310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4489.254192] Lustre: lustre-OST0000: new disk, initializing [ 4489.258373] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4489.263871] Lustre: 137951:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4491.251629] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4491.262322] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4491.457951] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4496.526821] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4507.332456] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4518.231681] Lustre: lustre-MDD0000: changelog on [ 4521.181276] Lustre: lustre-MDD0001: changelog on [ 6529.930868] Lustre: lustre-MDD0001: changelog off [ 6531.808541] Lustre: 135991:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6532.665412] Lustre: lustre-MDD0000: changelog off [ 6539.239514] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6539.248161] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6539.251246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6539.251254] Lustre: Skipped 4 previous similar messages [ 6539.255030] Lustre: Skipped 1 previous similar message [ 6544.363368] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6544.372207] Lustre: Skipped 2 previous similar messages [ 6545.224106] Lustre: server umount lustre-OST0000 complete [ 6549.482772] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6549.492129] Lustre: Skipped 1 previous similar message [ 6549.493625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6555.882516] Lustre: server umount lustre-MDT0000 complete [ 6559.722721] LustreError: 135991:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6559.739064] LustreError: 135991:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6559.973545] LustreError: 135976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788630916 with bad export cookie 9284191056285749610 [ 6559.984564] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6559.985701] LustreError: 135976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6560.608287] Lustre: server umount lustre-MDT0001 complete [ 6572.760674] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 6575.451742] Key type lgssc unregistered [ 6575.714635] LNet: 147111:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6575.721522] LNetError: 147111:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6576.745304] LNet: Removed LNI 192.168.203.123@tcp [ 6577.762183] Key type .llcrypt unregistered [ 6577.764121] Key type ._llcrypt unregistered [ 6600.750554] Key type ._llcrypt registered [ 6600.754299] Key type .llcrypt registered [ 6600.943991] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 6618.886378] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 6620.289366] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6620.441152] alg: No test for adler32 (adler32-zlib) [ 6621.370334] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [ 6621.552608] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 6623.447244] Key type lgssc registered [ 6625.065140] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6662.508217] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6662.559827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6663.859706] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6663.896715] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6664.006359] Lustre: lustre-MDT0000: new disk, initializing [ 6664.077380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6664.097402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6668.526530] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6679.801267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6679.898927] Lustre: 150306:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6679.953696] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6679.957550] Lustre: Skipped 1 previous similar message [ 6680.029463] Lustre: lustre-MDT0001: new disk, initializing [ 6680.162850] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6680.205418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6680.220643] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6684.504704] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6694.247398] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6699.889459] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6706.124841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6706.355391] Lustre: lustre-OST0000: new disk, initializing [ 6706.358584] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6706.364043] Lustre: 152272:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6706.442564] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6713.015456] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6716.533047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6716.552817] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6716.697687] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6725.172205] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6732.965777] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_module obdecho/obdecho [ 6735.583671] LustreError: 153278:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6735.595387] LustreError: 153278:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6738.556456] LustreError: 153375:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6752.742256] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6752.759880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6752.762387] Lustre: Skipped 1 previous similar message [ 6752.786682] Lustre: Skipped 2 previous similar messages [ 6757.859562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6757.875177] Lustre: Skipped 2 previous similar messages [ 6758.280128] Lustre: server umount lustre-MDT0000 complete [ 6762.392464] LustreError: 150299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788631119 with bad export cookie 18342957475975512310 [ 6762.399859] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6762.403668] LustreError: 150299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6762.777195] Lustre: server umount lustre-MDT0001 complete [ 6777.682847] Lustre: server umount lustre-OST0000 complete [ 6797.977798] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 6807.948965] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 6846.689305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6846.989466] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6847.078409] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6847.178467] Lustre: lustre-MDT0000: new disk, initializing [ 6847.333443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6847.354392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6852.638748] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6864.990389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6865.107544] Lustre: 157093:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6865.151086] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6865.155973] Lustre: Skipped 1 previous similar message [ 6865.225369] Lustre: lustre-MDT0001: new disk, initializing [ 6865.320881] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6865.376982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6865.396837] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6869.777636] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6879.679323] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6885.281485] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6891.681856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6891.943953] Lustre: lustre-OST0000: new disk, initializing [ 6891.949087] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6891.958338] Lustre: 159062:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6892.060547] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6898.338849] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6899.289487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6899.302961] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6899.427273] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6908.510185] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6914.326771] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6914.573402] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6919.588938] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6919.857111] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6925.285794] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6925.298757] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6925.301975] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6925.318661] Lustre: Skipped 2 previous similar messages [ 6925.333531] Lustre: Skipped 1 previous similar message [ 6928.036361] Lustre: server umount lustre-OST0000 complete [ 6931.937130] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6931.949483] LustreError: Skipped 1 previous similar message [ 6931.957658] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6931.980092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6935.522318] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6938.162123] Lustre: server umount lustre-MDT0000 complete [ 6941.829219] LustreError: 157085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788631298 with bad export cookie 18342957475975513269 [ 6941.831866] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6941.843908] LustreError: 157085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6942.191168] Lustre: server umount lustre-MDT0001 complete [ 6945.944830] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 14:01:40 (1788631300) [ 6947.810724] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6949.481692] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 14:01:44 (1788631304) [ 6957.762424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6958.294906] LustreError: 161486:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6958.317248] LustreError: 161486:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6958.403443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6963.097423] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6963.690408] LustreError: 161487:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6968.805976] LustreError: 161486:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6971.769107] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6977.394161] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6985.735240] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6991.980592] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6999.270543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6999.628370] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6999.637744] Lustre: Skipped 1 previous similar message [ 7006.665357] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7014.646717] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7025.514587] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.23@tcp (stopping) [ 7025.520525] Lustre: Skipped 2 previous similar messages [ 7026.577888] Lustre: server umount lustre-MDT0000 complete [ 7027.687986] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7027.701510] LustreError: 161486:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7027.703201] Lustre: Skipped 1 previous similar message [ 7027.748236] LustreError: 161486:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7030.485742] LustreError: 161465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788631387 with bad export cookie 18342957475975514151 [ 7030.499127] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7030.507160] LustreError: 161465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7032.802188] LustreError: 161481:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7032.828152] LustreError: 161481:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7035.732925] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.23@tcp (stopping) [ 7035.749508] Lustre: Skipped 2 previous similar messages [ 7036.894104] Lustre: server umount lustre-MDT0001 complete [ 7043.444768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7056.234517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7056.477838] LustreError: 165329:0:(llog.c:1655:llog_backup()) MGC192.168.203.123@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7056.487607] Lustre: 165329:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.123@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7060.448971] LustreError: 165334:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7060.467311] LustreError: 165334:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7077.715747] LustreError: 165336:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7077.745583] LustreError: 165336:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 7111.658207] LustreError: 165336:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7111.684105] LustreError: 165336:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 24 previous similar messages [ 7117.279315] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7117.289027] Lustre: 165329:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.123@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7158.274970] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7162.509661] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7163.370076] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7171.011100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7175.648561] LustreError: 148530:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f59e9cf1 x1875515516519552/t0(0) o250->MGC192.168.203.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7175.668572] LustreError: 165335:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7175.683637] LustreError: 165335:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 45 previous similar messages [ 7180.188829] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7181.301475] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7186.847480] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7188.136567] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7192.939355] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7194.569831] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7216.329061] Lustre: server umount lustre-MDT0000 complete [ 7217.120030] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7217.129800] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7217.148356] Lustre: Skipped 4 previous similar messages [ 7219.371319] LustreError: 161466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788631576 with bad export cookie 18342957475975520605 [ 7219.377789] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7219.388130] LustreError: 161466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7219.722287] Lustre: server umount lustre-MDT0001 complete [ 7234.254824] Lustre: server umount lustre-OST0000 complete [ 7255.901049] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 7267.381569] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 7303.306230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7303.599339] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7303.621843] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7303.723306] Lustre: lustre-MDT0000: new disk, initializing [ 7303.797637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7303.801013] Lustre: Skipped 1 previous similar message [ 7303.825230] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7308.566940] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7318.614414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7318.671857] Lustre: 170433:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7318.940939] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7324.349283] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7335.531107] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7341.584695] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7348.412463] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7348.710768] Lustre: lustre-OST0000: new disk, initializing [ 7348.713489] Lustre: Skipped 1 previous similar message [ 7348.718155] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7348.725172] Lustre: Skipped 2 previous similar messages [ 7348.730963] Lustre: 172403:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7354.880932] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7355.448043] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7355.465219] Lustre: Skipped 1 previous similar message [ 7355.484361] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7355.564093] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7363.662997] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7368.004928] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7368.189408] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7372.938275] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7373.112892] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7376.352563] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7376.357298] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7376.363127] LustreError: Skipped 1 previous similar message [ 7376.364161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7376.364167] Lustre: Skipped 1 previous similar message [ 7376.390464] Lustre: Skipped 1 previous similar message [ 7380.898581] Lustre: server umount lustre-OST0000 complete [ 7385.570788] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7390.408995] Lustre: server umount lustre-MDT0000 complete [ 7391.720636] LustreError: 170445:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7391.736454] LustreError: 170445:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 7393.928631] LustreError: 170426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788631750 with bad export cookie 18342957475975537440 [ 7393.930286] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7393.954707] LustreError: 170426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7394.280816] Lustre: server umount lustre-MDT0001 complete [ 7401.872744] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 14:09:17 (1788631757) [ 7407.628483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7411.935930] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7420.496713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7425.892875] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7433.661404] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7440.526992] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7448.656739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7457.241731] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7467.342793] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7478.174607] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 14:10:32 (1788631832) [ 7483.362444] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7483.382844] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7483.396526] Lustre: Skipped 2 previous similar messages [ 7483.403485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7483.407737] Lustre: Skipped 4 previous similar messages [ 7488.736939] Lustre: server umount lustre-OST0000 complete [ 7499.352437] Lustre: server umount lustre-MDT0000 complete [ 7504.345857] LustreError: 174793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788631861 with bad export cookie 18342957475975538322 [ 7504.368109] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7505.015736] Lustre: server umount lustre-MDT0001 complete [ 7517.627585] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 7520.376046] Key type lgssc unregistered [ 7520.609546] LNet: 178541:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7520.626294] LNetError: 178541:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7520.642955] LNet: Removed LNI 192.168.203.123@tcp [ 7521.544212] Key type .llcrypt unregistered [ 7521.549445] Key type ._llcrypt unregistered [ 7540.396341] Key type ._llcrypt registered [ 7540.399290] Key type .llcrypt registered [ 7540.487636] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 7541.580199] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7541.622083] alg: No test for adler32 (adler32-zlib) [ 7542.689354] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [ 7542.953858] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 7544.663151] Key type lgssc registered [ 7545.732646] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7554.006770] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7554.028477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7569.567442] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7574.687838] LustreError: 179781:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.123@tcp: failed processing log, type 4: rc = -110 [ 7579.871458] LustreError: 179781:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7579.892862] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7579.901602] LustreError: MGC192.168.203.123@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7579.922268] LustreError: 179781:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7579.938547] LustreError: 179781:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -22 [ 7579.964723] LustreError: 179781:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 7580.042284] Lustre: server umount lustre-OST0000 complete [ 7580.045970] LustreError: 179781:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7588.603834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7590.047103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7594.300152] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7599.588990] LustreError: 180278:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7599.607616] LustreError: 180278:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7602.761161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7603.072872] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7607.635394] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7614.928329] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7619.926328] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7626.428362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7626.597180] LustreError: 182093:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7626.611262] Lustre: 182093:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.203.123@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7626.733544] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7631.919636] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7640.740683] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7644.132948] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7644.155705] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7644.168579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7645.152737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7645.154088] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7645.156795] Lustre: Skipped 1 previous similar message [ 7645.166231] Lustre: Skipped 1 previous similar message [ 7649.275736] Lustre: server umount lustre-MDT0000 complete [ 7650.277787] LustreError: 180279:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7650.303182] LustreError: 180279:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7652.745252] LustreError: 180259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788632009 with bad export cookie 14255452607342430384 [ 7652.756813] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7652.991255] Lustre: server umount lustre-MDT0001 complete [ 7667.496587] Lustre: server umount lustre-OST0000 complete [ 7679.104550] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 7691.448814] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 7729.708176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7729.975264] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7730.013491] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7730.098290] Lustre: lustre-MDT0000: new disk, initializing [ 7730.170820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7730.185792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7734.600823] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7746.442616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7746.545545] Lustre: 185621:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7746.627918] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7746.630705] Lustre: Skipped 1 previous similar message [ 7746.716255] Lustre: lustre-MDT0001: new disk, initializing [ 7746.804296] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7746.862906] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7746.873745] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7751.389654] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7761.865073] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7768.003857] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7775.622074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7776.117625] Lustre: lustre-OST0000: new disk, initializing [ 7776.125408] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7776.138793] Lustre: 187590:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7776.230756] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7783.017907] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7786.084433] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7786.098182] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7786.262591] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7793.694504] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7798.797167] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7799.020407] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7804.313834] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7804.545596] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7806.952354] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7806.953626] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7806.962767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7806.973524] Lustre: Skipped 1 previous similar message [ 7806.981059] Lustre: Skipped 1 previous similar message [ 7812.072594] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7812.075431] Lustre: Skipped 1 previous similar message [ 7812.480804] Lustre: server umount lustre-OST0000 complete [ 7817.185955] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7817.213047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7817.219841] Lustre: Skipped 1 previous similar message [ 7823.162388] Lustre: server umount lustre-MDT0000 complete [ 7826.872354] LustreError: 185613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788632183 with bad export cookie 14255452607342431105 [ 7826.881236] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7826.885732] LustreError: 185613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 7827.175780] Lustre: server umount lustre-MDT0001 complete [ 7836.142857] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 14:16:31 (1788632191) [ 7853.041569] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 7855.595737] Key type lgssc unregistered [ 7855.835773] LNet: 190533:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7855.841283] LNetError: 190533:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7855.859991] LNet: Removed LNI 192.168.203.123@tcp [ 7856.643280] Key type .llcrypt unregistered [ 7856.645691] Key type ._llcrypt unregistered [ 7874.036622] Key type ._llcrypt registered [ 7874.039771] Key type .llcrypt registered [ 7874.130614] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 7875.289084] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7875.325727] alg: No test for adler32 (adler32-zlib) [ 7876.362334] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [ 7876.548707] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 7878.199621] Key type lgssc registered [ 7879.142900] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7887.266640] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7887.287894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7888.778207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7893.351990] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7899.109232] LustreError: 191746:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7899.134273] LustreError: 191746:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7902.152849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7902.453966] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7906.802703] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7915.908717] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7922.482767] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7930.855334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7931.050839] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7931.053974] LustreError: 193611:0:(llog.c:1696:llog_backup()) MGC192.168.203.123@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7931.063574] Lustre: 193611:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.203.123@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7931.210126] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7937.155688] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7944.661820] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7947.617867] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7947.622958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7947.632545] Lustre: Skipped 1 previous similar message [ 7947.648893] Lustre: Skipped 2 previous similar messages [ 7952.870327] LustreError: 191746:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7952.882055] LustreError: 191746:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7952.945161] Lustre: server umount lustre-MDT0000 complete [ 7956.468458] LustreError: 191725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788632313 with bad export cookie 10218720028584091509 [ 7956.474109] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7956.480962] LustreError: 191725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7957.919434] LustreError: 192476:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000007b8d332d x1875516832378368/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7958.000908] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7958.001969] LustreError: 191742:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7958.017832] Lustre: Skipped 1 previous similar message [ 7958.053795] LustreError: 191742:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7958.227695] Lustre: server umount lustre-MDT0001 complete [ 7962.401947] Lustre: server umount lustre-OST0000 complete [ 7972.265690] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 7983.293833] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 8022.528626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8022.783170] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8022.808301] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8022.884131] Lustre: lustre-MDT0000: new disk, initializing [ 8022.956746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8022.987583] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8027.811292] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8038.091158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8038.202491] Lustre: 197135:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8038.257382] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8038.259351] Lustre: Skipped 1 previous similar message [ 8038.262798] LustreError: 197920:0:(llog.c:1696:llog_backup()) MGC192.168.203.123@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8038.278291] LustreError: 197920:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8038.283719] Lustre: 197920:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.123@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8038.301891] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8038.307523] Lustre: Skipped 1 previous similar message [ 8038.450642] Lustre: lustre-MDT0001: new disk, initializing [ 8038.599836] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8038.645685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8038.660242] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8044.043585] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8055.732445] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8062.625845] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8070.324634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8070.555355] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8070.557850] Lustre: Skipped 1 previous similar message [ 8070.561180] LustreError: 199085:0:(llog.c:1696:llog_backup()) MGC192.168.203.123@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8070.566881] LustreError: 199085:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8070.572464] Lustre: 199085:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.123@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8070.581489] Lustre: 199085:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8070.600695] Lustre: lustre-OST0000: new disk, initializing [ 8070.604195] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8070.612347] Lustre: 199110:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8070.702601] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8076.333237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8076.346184] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8076.439866] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8076.581607] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8085.188128] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8090.294782] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8090.552964] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8096.404758] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8096.734115] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8101.878136] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8101.904967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8101.918846] Lustre: Skipped 1 previous similar message [ 8105.686950] Lustre: server umount lustre-OST0000 complete [ 8110.561547] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8110.570245] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8110.580793] Lustre: Skipped 1 previous similar message [ 8110.593336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8116.719899] Lustre: server umount lustre-MDT0000 complete [ 8117.222422] LustreError: 197148:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8117.253397] LustreError: 197148:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8120.358545] LustreError: 199109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788632477 with bad export cookie 10218720028584092230 [ 8120.360921] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8120.372349] LustreError: 199109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8120.718923] Lustre: server umount lustre-MDT0001 complete [ 8130.974437] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 14:21:25 (1788632485) [ 8149.275702] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 8159.495539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8160.031838] LustreError: 202237:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8160.092567] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8164.807167] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8169.445951] LustreError: 202238:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8169.486083] LustreError: 202238:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8173.593411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8178.496331] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8181.800267] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8189.025177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8196.779715] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8205.003880] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8205.181371] Lustre: lustre-OST0001: new disk, initializing [ 8205.184528] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8205.190179] Lustre: 204545:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8205.281768] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8205.286631] Lustre: Skipped 2 previous similar messages [ 8211.924895] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8213.549569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8213.579347] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8213.684064] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8224.290502] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8231.791389] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8233.754876] Lustre: DEBUG MARKER: ADD OST3 [ 8239.015077] loop: module loaded [ 8239.117434] loop0: detected capacity change from 0 to 204800000 [ 8239.364581] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8239.382842] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8239.393943] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8245.957367] loop0: detected capacity change from 0 to 204800000 [ 8247.156960] Lustre: DEBUG MARKER: STOP OST3 [ 8252.068275] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8252.306572] Lustre: lustre-OST0003: new disk, initializing [ 8252.313536] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8252.327292] Lustre: 206542:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8254.027048] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8254.176183] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8254.193491] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8254.247208] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8256.095390] LustreError: 206717:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8256.229476] Lustre: Failing over lustre-OST0003 [ 8256.488963] Lustre: server umount lustre-OST0003 complete [ 8258.933838] LustreError: 203725:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8259.389283] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8259.554560] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8259.562238] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8259.573133] Lustre: Skipped 1 previous similar message [ 8267.617570] Lustre: DEBUG MARKER: START OST3 again [ 8274.746321] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8274.919902] Lustre: lustre-OST0003: Not available for connect from 0@lo (not set up) [ 8274.933932] Lustre: Skipped 3 previous similar messages [ 8275.140346] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8275.155289] Lustre: Skipped 1 previous similar message [ 8275.173464] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8276.469687] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8277.119401] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8277.120892] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8281.224565] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8303.201381] Lustre: Failing over lustre-OST0003 [ 8303.276961] Lustre: server umount lustre-OST0003 complete [ 8306.166963] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8306.187703] Lustre: Skipped 1 previous similar message [ 8306.199718] LustreError: 203725:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8306.235566] LustreError: 203725:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 8322.528434] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8322.533782] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8322.550344] Lustre: Skipped 1 previous similar message [ 8322.560941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8331.750630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8331.766472] Lustre: Skipped 7 previous similar messages [ 8336.351792] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8336.547883] Lustre: server umount lustre-MDT0000 complete [ 8339.909091] LustreError: 202219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788632696 with bad export cookie 10218720028584093112 [ 8339.913835] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8339.922177] LustreError: 202219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8350.115304] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8350.122318] Lustre: Skipped 5 previous similar messages [ 8354.271256] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8354.529798] Lustre: server umount lustre-MDT0001 complete [ 8358.452989] Lustre: server umount lustre-OST0000 complete [ 8362.045577] Lustre: server umount lustre-OST0001 complete [ 8368.832609] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 8377.640808] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 8421.660045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8422.010313] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8422.067836] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8422.173837] Lustre: lustre-MDT0000: new disk, initializing [ 8422.283577] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8422.312145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8428.354244] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8441.630919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8441.787087] Lustre: 212002:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8441.873616] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8441.881338] Lustre: Skipped 1 previous similar message [ 8442.047602] Lustre: lustre-MDT0001: new disk, initializing [ 8442.219467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8442.231321] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8447.555190] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8458.886520] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8465.755533] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8474.186572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8474.512562] Lustre: lustre-OST0000: new disk, initializing [ 8474.516456] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8474.526524] Lustre: 213976:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8476.270406] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8476.296276] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8476.477219] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8481.709106] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8492.044308] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8497.816108] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8498.006921] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8502.768107] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8503.139863] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8507.369172] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8507.393721] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8507.407726] Lustre: Skipped 5 previous similar messages [ 8507.425997] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8507.432190] Lustre: Skipped 1 previous similar message [ 8511.515363] Lustre: server umount lustre-OST0000 complete [ 8521.959037] Lustre: server umount lustre-MDT0000 complete [ 8522.723883] LustreError: 213952:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8522.753362] LustreError: 213952:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 20 previous similar messages [ 8526.495853] LustreError: 211997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788632883 with bad export cookie 10218720028584095408 [ 8526.506112] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8526.525787] LustreError: 211997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8526.951484] Lustre: server umount lustre-MDT0001 complete [ 8531.802621] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 14:28:06 (1788632886) [ 8551.207187] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 8560.790811] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 8596.927625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8597.145331] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8597.175911] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8597.271287] Lustre: lustre-MDT0000: new disk, initializing [ 8597.355205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8601.991213] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8612.737780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8612.872182] Lustre: 218588:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8613.245972] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8617.541454] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8626.967539] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8633.565788] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8641.138537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8641.434582] Lustre: 220556:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8642.692938] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8648.640874] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8659.087523] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8664.236692] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8664.505568] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8670.066831] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8670.252383] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8673.251720] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8673.252242] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8673.263397] LustreError: Skipped 1 previous similar message [ 8673.266672] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8673.276112] Lustre: Skipped 4 previous similar messages [ 8673.283940] Lustre: Skipped 2 previous similar messages [ 8678.625397] Lustre: server umount lustre-OST0000 complete [ 8689.480896] Lustre: server umount lustre-MDT0000 complete [ 8693.141891] LustreError: 218582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788633050 with bad export cookie 10218720028584096290 [ 8693.144144] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8693.150864] LustreError: 218582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8693.531700] Lustre: server umount lustre-MDT0001 complete [ 8701.995721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8702.455506] LustreError: 222775:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8702.523561] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8702.529129] Lustre: Skipped 5 previous similar messages [ 8707.713506] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8717.710062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8723.479525] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8732.632251] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8739.824587] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8746.618901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8753.882262] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8762.932726] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8774.125588] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8780.099888] Lustre: server umount lustre-OST0000 complete [ 8790.838667] Lustre: server umount lustre-MDT0000 complete [ 8795.035770] LustreError: 222756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788633151 with bad export cookie 10218720028584097172 [ 8795.046047] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8795.052896] LustreError: 222756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8795.528289] Lustre: server umount lustre-MDT0001 complete [ 8902.393711] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 8905.599702] Key type lgssc unregistered [ 8905.925519] LNet: 226442:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8905.940809] LNetError: 226442:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8905.965879] LNet: Removed LNI 192.168.203.123@tcp [ 8906.955194] Key type .llcrypt unregistered [ 8906.962824] Key type ._llcrypt unregistered [ 8933.982677] Key type ._llcrypt registered [ 8933.987903] Key type .llcrypt registered [ 8934.141977] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 8950.447255] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 8951.334857] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8951.509048] alg: No test for adler32 (adler32-zlib) [ 8952.691976] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [ 8952.986691] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 8954.703241] Key type lgssc registered [ 8955.851775] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8993.636157] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8993.663453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8994.954973] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8994.978423] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8995.063849] Lustre: lustre-MDT0000: new disk, initializing [ 8995.166434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8995.182384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8999.966609] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 9011.281233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9011.444718] Lustre: 229789:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9011.576595] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9011.582724] Lustre: Skipped 1 previous similar message [ 9011.697646] Lustre: lustre-MDT0001: new disk, initializing [ 9011.824885] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9011.870520] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9011.897432] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9016.868479] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 9026.613727] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9032.573441] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9039.990033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9040.348304] Lustre: lustre-OST0000: new disk, initializing [ 9040.362182] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9040.370368] Lustre: 231757:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9040.464064] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9047.376349] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 9049.203071] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9049.226773] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9049.362138] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9059.168388] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9065.081653] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9065.288915] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9070.441963] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9070.739627] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9075.175632] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9075.181354] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9075.205819] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9075.226285] LustreError: Skipped 1 previous similar message [ 9079.010623] Lustre: server umount lustre-OST0000 complete [ 9083.367883] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9083.374950] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9083.388610] Lustre: Skipped 1 previous similar message [ 9083.402260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9083.408016] Lustre: Skipped 1 previous similar message [ 9085.422550] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9085.433196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9085.463094] Lustre: Skipped 1 previous similar message [ 9091.276082] Lustre: server umount lustre-MDT0000 complete [ 9095.390142] LustreError: 229782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788633452 with bad export cookie 3154089758794469794 [ 9095.399587] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9095.405359] LustreError: 229782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9095.726644] Lustre: server umount lustre-MDT0001 complete [ 9117.127132] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 9127.370682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9127.779042] LustreError: 234691:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9127.803896] LustreError: 234691:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9127.854605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9132.936228] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 9133.024538] LustreError: 234692:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9137.127875] LustreError: 234691:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9142.254776] LustreError: 234692:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9142.461519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9142.966844] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9147.378607] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 9151.003620] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9158.528758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9158.998124] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9166.211458] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 9176.487397] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9176.745239] Lustre: lustre-OST0001: new disk, initializing [ 9176.753416] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9176.763880] Lustre: 236999:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9176.886856] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9182.401954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9182.424066] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9182.542346] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9184.148552] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 9196.253518] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9205.277543] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9221.788498] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 14:39:36 (1788633576) [ 9224.672662] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9224.681613] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9224.697357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9228.775748] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9228.785289] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9228.802073] Lustre: Skipped 2 previous similar messages [ 9228.806579] Lustre: Skipped 2 previous similar messages [ 9230.842185] Lustre: server umount lustre-MDT0000 complete [ 9233.893290] LustreError: 236592:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9233.935196] LustreError: 236592:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9235.436404] LustreError: 234673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788633592 with bad export cookie 3154089758794470676 [ 9235.471218] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9235.826169] Lustre: server umount lustre-MDT0001 complete [ 9250.623704] Lustre: server umount lustre-OST0000 complete [ 9265.073022] Lustre: server umount lustre-OST0001 complete [ 9273.018505] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 9285.080662] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 9324.324593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9324.618316] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9324.645800] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9324.743382] Lustre: lustre-MDT0000: new disk, initializing [ 9324.871687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9324.912384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9329.445798] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 9340.532420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9340.714379] Lustre: 240950:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9340.811910] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9340.819649] Lustre: Skipped 1 previous similar message [ 9340.948394] Lustre: lustre-MDT0001: new disk, initializing [ 9341.121151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9341.137827] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9346.311292] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 9355.873651] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9362.407539] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9370.270135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9370.744545] Lustre: lustre-OST0000: new disk, initializing [ 9370.751250] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9370.760603] Lustre: 242917:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9372.239237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9372.255175] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9372.401964] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9377.849239] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 9390.069779] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9399.123591] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9399.425686] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9404.640934] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9404.905253] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9408.489199] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9408.494654] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9408.500251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9408.538405] Lustre: Skipped 1 previous similar message [ 9421.280644] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9421.540922] Lustre: server umount lustre-OST0000 complete [ 9428.448926] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9428.459257] LustreError: Skipped 1 previous similar message [ 9428.462201] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9428.476826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9428.497431] Lustre: Skipped 5 previous similar messages [ 9431.849578] Lustre: server umount lustre-MDT0000 complete [ 9434.083345] LustreError: 240961:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9434.098491] LustreError: 240961:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9436.284538] LustreError: 240942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788633793 with bad export cookie 3154089758794472811 [ 9436.298440] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9436.304187] LustreError: 240942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 9436.743751] Lustre: server umount lustre-MDT0001 complete [ 9446.361988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9447.064583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9447.070421] Lustre: Skipped 2 previous similar messages [ 9451.942803] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 9452.516860] LustreError: 245135:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9452.532978] LustreError: 245135:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9461.988548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9468.010738] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 9476.903896] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9484.405462] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9492.382384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9500.227571] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 9508.720466] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9655.780114] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9655.791112] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9655.802204] Lustre: Skipped 1 previous similar message [ 9655.810534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9655.818404] Lustre: Skipped 1 previous similar message [ 9669.089077] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9669.234848] Lustre: server umount lustre-OST0000 complete [ 9679.833092] Lustre: server umount lustre-MDT0000 complete [ 9680.354458] LustreError: 245136:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9680.384841] LustreError: 245136:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9685.131902] LustreError: 245118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788634042 with bad export cookie 3154089758794473686 [ 9685.146775] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9685.146975] LustreError: 245118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9685.942133] Lustre: server umount lustre-MDT0001 complete [ 9778.760825] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 9782.427040] Key type lgssc unregistered [ 9782.888991] LNet: 248742:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9782.896955] LNetError: 248742:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9782.922613] LNet: Removed LNI 192.168.203.123@tcp [ 9784.085881] Key type .llcrypt unregistered [ 9784.087808] Key type ._llcrypt unregistered [ 9822.240688] Key type ._llcrypt registered [ 9822.249773] Key type .llcrypt registered [ 9822.426855] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 9838.824779] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 9839.952260] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9840.026436] alg: No test for adler32 (adler32-zlib) [ 9841.129570] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [ 9841.489565] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 9843.183173] Key type lgssc registered [ 9844.610245] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9895.387897] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 9906.327291] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9906.379734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9907.625033] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9907.663456] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9907.784200] Lustre: lustre-MDT0000: new disk, initializing [ 9907.929870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9907.950609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9912.918076] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 9924.249072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9924.453840] Lustre: 252817:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9924.534499] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9924.542592] Lustre: Skipped 1 previous similar message [ 9924.627491] Lustre: lustre-MDT0001: new disk, initializing [ 9924.724775] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9924.760276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9924.778883] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9929.564941] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 9934.801770] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9942.251965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9942.552531] Lustre: lustre-OST0000: new disk, initializing [ 9942.557866] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9942.573931] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 9942.575500] Lustre: 254452:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9942.710189] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9948.175291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9948.184556] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9948.221410] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9949.561337] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 9960.739135] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9960.936417] Lustre: lustre-OST0001: new disk, initializing [ 9960.940414] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9960.946666] Lustre: 255321:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9961.052774] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9967.822807] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 9969.775457] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9969.785052] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9969.919989] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9980.670074] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9991.994859] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10005.984246] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10005.994511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10006.017724] Lustre: Skipped 2 previous similar messages [10006.032628] Lustre: Skipped 1 previous similar message [10011.105798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10011.117769] Lustre: Skipped 5 previous similar messages [10016.228783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10016.237962] Lustre: Skipped 2 previous similar messages [10019.295130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10019.548901] Lustre: server umount lustre-MDT0000 complete [10023.492486] LustreError: 252809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788634380 with bad export cookie 10201311764289262022 [10023.510865] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10023.527713] LustreError: 252809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10024.095047] Lustre: server umount lustre-MDT0001 complete [10039.061661] Lustre: server umount lustre-OST0000 complete [10042.658324] Lustre: 250320:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788634383/real 1788634383] req@ffff94d584580a80 x1875518892886400/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788634399 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10042.713510] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10042.732227] Lustre: Skipped 1 previous similar message [10043.477225] Lustre: server umount lustre-OST0001 complete [10060.001979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10071.608709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10081.813634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10093.367039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10116.219840] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [10127.435989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10127.457768] Lustre: lustre-MDT0000: reset Object Index mappings [10127.909733] LustreError: 260335:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10127.933417] LustreError: 260335:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10127.998355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10133.042868] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10133.477116] LustreError: 260336:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10137.579773] LustreError: 260335:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10141.906852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10141.942261] Lustre: lustre-MDT0001: reset Object Index mappings [10142.422786] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10146.284754] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10149.009830] Lustre: 261477:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10155.508432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10162.305592] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10167.148436] LustreError: 261834:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10170.922492] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10171.224370] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10171.230073] Lustre: Skipped 1 previous similar message [10172.316951] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10172.343511] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10179.016589] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10187.470515] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10192.513477] Lustre: 263353:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10197.991744] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10197.997977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10198.004610] Lustre: Skipped 2 previous similar messages [10198.008750] Lustre: Skipped 2 previous similar messages [10199.884017] LustreError: 263455:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x736d61726170:0x3:0x0].0x0 (ffff94d584106400) refcount nonzero (2) after lock cleanup; forcing cleanup. [10199.961870] Lustre: server umount lustre-MDT0000 complete [10203.116148] LustreError: 260331:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10203.136521] LustreError: 260331:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10204.028888] LustreError: 261479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788634560 with bad export cookie 10201311764289268805 [10204.032971] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10204.052511] LustreError: 261479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10204.628061] Lustre: server umount lustre-MDT0001 complete [10218.879264] Lustre: server umount lustre-OST0000 complete [10234.527463] Lustre: server umount lustre-OST0001 complete [10258.107705] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 14:56:52 (1788634612) [10276.504925] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [10288.453206] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [10331.399269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10331.650321] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10331.692121] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10331.795297] Lustre: lustre-MDT0000: new disk, initializing [10331.875602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10331.898697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10336.997877] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10348.682332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10348.798755] Lustre: 267607:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10348.861855] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10348.865899] Lustre: Skipped 1 previous similar message [10348.951348] Lustre: lustre-MDT0001: new disk, initializing [10349.072880] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10349.088822] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10354.243393] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10364.127401] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10371.321984] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10379.550940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10379.908632] Lustre: lustre-OST0000: new disk, initializing [10379.913085] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10379.931269] Lustre: 269579:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10381.958191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10381.974762] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10382.079377] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10386.682240] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10395.888136] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10400.991756] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10401.242274] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10405.606945] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10405.789638] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10407.907056] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10407.914374] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10407.918347] Lustre: Skipped 1 previous similar message [10407.924540] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10407.946880] Lustre: Skipped 3 previous similar messages [10422.241827] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10422.626698] Lustre: server umount lustre-OST0000 complete [10428.385400] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10428.403399] Lustre: Skipped 1 previous similar message [10428.409120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10428.423384] Lustre: Skipped 5 previous similar messages [10432.911696] Lustre: server umount lustre-MDT0000 complete [10433.505143] LustreError: 267620:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10433.531254] LustreError: 267620:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10436.724982] LustreError: 269578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788634793 with bad export cookie 10201311764289270905 [10436.727843] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10436.739857] LustreError: 269578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10437.090791] Lustre: server umount lustre-MDT0001 complete [10456.635745] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [10467.571853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10468.183833] LustreError: 272517:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10468.229060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10468.237512] Lustre: Skipped 2 previous similar messages [10473.252684] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10481.727526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10487.273171] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10490.479142] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10497.250729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10504.561395] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10515.213054] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10515.420966] Lustre: lustre-OST0001: new disk, initializing [10515.427210] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10515.433146] Lustre: 274838:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10517.293224] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10517.306904] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10517.413676] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10522.691628] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10533.531726] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10537.946696] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10543.446967] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10543.498175] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10558.440872] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10558.447209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10558.469488] Lustre: Skipped 4 previous similar messages [10558.476288] Lustre: Skipped 2 previous similar messages [10560.476854] Lustre: server umount lustre-MDT0000 complete [10563.571430] LustreError: 272517:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10563.599962] LustreError: 272517:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10564.147783] LustreError: 272497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788634921 with bad export cookie 10201311764289271787 [10564.153197] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10564.161313] LustreError: 272497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10564.558735] Lustre: server umount lustre-MDT0001 complete [10579.076776] Lustre: server umount lustre-OST0000 complete [10592.476294] Lustre: server umount lustre-OST0001 complete [10610.691924] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [10622.435609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10622.947196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10622.955914] Lustre: Skipped 3 previous similar messages [10628.035406] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10628.066252] LustreError: 278189:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10628.081910] LustreError: 278189:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10638.408206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10644.201816] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10648.141137] Lustre: 279328:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10655.472106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10663.391900] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10667.388789] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10673.367206] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10678.815126] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10680.806536] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10689.355894] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10693.525066] Lustre: 281206:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10699.521598] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10699.601968] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10713.266738] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 15:04:28 (1788635068) [10729.958235] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10729.958872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10729.966855] Lustre: Skipped 2 previous similar messages [10729.971091] Lustre: Skipped 2 previous similar messages [10736.798785] Lustre: server umount lustre-MDT0000 complete [10740.535965] LustreError: 278169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788635097 with bad export cookie 10201311764289278668 [10740.538373] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10740.549775] LustreError: 278169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10747.127556] Lustre: server umount lustre-MDT0001 complete [10766.303145] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10766.454078] Lustre: server umount lustre-OST0000 complete [10771.247443] Lustre: server umount lustre-OST0001 complete [10779.329231] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [10790.774472] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [10837.773887] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [10849.837853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10850.185655] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10850.277811] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10850.416756] Lustre: lustre-MDT0000: new disk, initializing [10850.567964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10855.982539] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10866.619439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10866.784092] Lustre: 285738:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10867.033736] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10872.091657] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10884.020588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10884.344171] Lustre: 287371:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10884.455028] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10884.466116] Lustre: Skipped 5 previous similar messages [10886.266969] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10891.617801] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10902.140798] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10902.344952] Lustre: 288242:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10904.420809] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10904.432085] Lustre: Skipped 1 previous similar message [10904.571391] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10908.611534] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10918.115087] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10922.063866] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10922.068098] Lustre: Skipped 1 previous similar message [10936.538685] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 15:08:11 (1788635291) [10938.851190] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10938.868575] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10938.882943] Lustre: Skipped 2 previous similar messages [10938.891431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10938.901722] Lustre: Skipped 4 previous similar messages [10940.880426] Lustre: server umount lustre-MDT0000 complete [10944.666489] LustreError: 285730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788635301 with bad export cookie 10201311764289286186 [10944.684063] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10944.691141] LustreError: 285730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10945.094757] Lustre: server umount lustre-MDT0001 complete [10958.963995] Lustre: server umount lustre-OST0000 complete [10961.703070] Lustre: 250321:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788635302/real 1788635302] req@ffff94d4853a4a80 x1875518893205760/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788635318 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10963.090282] Lustre: server umount lustre-OST0001 complete [10971.373561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10971.867496] LustreError: 290487:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10971.877373] LustreError: 290487:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 17 previous similar messages [10976.687469] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10984.965164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10989.950298] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10998.276409] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11005.144871] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11014.794057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11023.607935] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [11032.526901] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11050.393331] loop1: detected capacity change from 0 to 204800000 [11050.638737] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11050.651807] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11050.675960] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11055.871714] loop1: detected capacity change from 0 to 204800000 [11060.989991] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11061.070172] LustreError: 290477:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs32719: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11061.082522] LustreError: 290477:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs32719 [11061.087445] LustreError: 290477:0:(mgs_handler.c:517:mgs_target_reg()) fs32719-MDT0001: Error checking target index: rc = -13 [11061.102851] LustreError: fs32719-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11061.108894] LustreError: 294204:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -13 [11061.113906] LustreError: 294204:0:(tgt_mount.c:2132:server_put_super()) no obd fs32719-MDT0001 [11061.117973] LustreError: 294204:0:(tgt_mount.c:116:server_deregister_mount()) fs32719-MDT0001 not registered [11061.150839] Lustre: server umount fs32719-MDT0001 complete [11061.154353] LustreError: 294204:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11069.064805] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11069.277415] Lustre: Setting parameter fs32719-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs32719-MDT0001 [11069.357545] Lustre: srv-fs32719-MDT0001: No data found on store. Initialize space. [11069.369245] Lustre: Skipped 4 previous similar messages [11069.484056] Lustre: fs32719-MDT0001: new disk, initializing [11069.491491] Lustre: Skipped 3 previous similar messages [11069.605745] Lustre: srv-fs32719-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11069.618274] Lustre: 294672:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs32719-MDT0001-mdtlov: can't create new llog: rc = -115 [11069.633837] LustreError: 294672:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs32719-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11074.497767] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [11082.451556] Lustre: server umount fs32719-MDT0001 complete [11096.923996] loop1: detected capacity change from 0 to 204800000 [11097.590166] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11102.160085] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11102.530081] Lustre: 296243:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs32719-MDT0001-mdtlov: can't create new llog: rc = -115 [11102.541694] LustreError: 296243:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs32719-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11106.684975] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [11111.526687] Lustre: server umount fs32719-MDT0001 complete [11112.952933] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11123.183120] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11123.218254] Lustre: Skipped 6 previous similar messages [11129.680985] Lustre: server umount lustre-MDT0000 complete [11140.417332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11140.507738] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11146.233720] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11147.064777] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [11156.198558] loop1: detected capacity change from 0 to 204800000 [11162.157331] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11162.624351] Lustre: 298692:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs32719-MDT0001-mdtlov: can't create new llog: rc = -115 [11162.631751] LustreError: 298692:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs32719-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11167.114901] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [11172.217271] Lustre: server umount fs32719-MDT0001 complete [11194.852089] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11197.417831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11197.434727] Lustre: Skipped 13 previous similar messages [11199.806970] Lustre: server umount lustre-OST0000 complete [11207.651377] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11207.660392] LustreError: Skipped 1 previous similar message [11208.953946] Lustre: server umount lustre-MDT0000 complete [11212.362592] LustreError: 290468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788635569 with bad export cookie 10201311764289288867 [11212.367791] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11212.385182] LustreError: 290468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11212.758073] Lustre: server umount lustre-MDT0001 complete [11223.094225] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [11226.095914] Key type lgssc unregistered [11226.359686] LNet: 301032:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11226.367554] LNetError: 301032:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11226.376524] LNet: Removed LNI 192.168.203.123@tcp [11227.199128] Key type .llcrypt unregistered [11227.201850] Key type ._llcrypt unregistered [11247.816382] Key type ._llcrypt registered [11247.818432] Key type .llcrypt registered [11247.909668] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [11249.389080] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11249.452246] alg: No test for adler32 (adler32-zlib) [11250.457644] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [11250.700581] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [11252.359216] Key type lgssc registered [11253.358739] Lustre: Echo OBD driver; http://www.lustre.org/ [11261.446950] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11261.476985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11263.273128] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11268.558376] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [11273.699801] LustreError: 302308:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11273.719473] LustreError: 302308:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11276.650897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11277.037744] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11281.493967] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [11288.308328] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11293.152826] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11300.101262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11300.498368] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11301.551215] LustreError: 304135:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11306.993048] LustreError: 304216:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11307.207468] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [11309.047876] LustreError: 304136:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11309.102035] LustreError: 304136:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11314.179224] LustreError: 304136:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11314.223187] LustreError: 304136:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11320.225516] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11322.684419] LustreError: 304138:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11322.725602] LustreError: 304138:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11339.760354] LustreError: 304216:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11339.778521] LustreError: 304216:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [11342.515507] loop1: detected capacity change from 0 to 204800000 [11342.749205] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11342.763251] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11342.805738] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11348.419830] loop1: detected capacity change from 0 to 204800000 [11354.427141] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11354.643980] Lustre: fs11605-OST0000: new disk, initializing [11354.648980] Lustre: srv-fs11605-OST0000: No data found on store. Initialize space. [11354.658844] Lustre: 306041:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11354.752776] Lustre: fs11605-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11360.785248] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [11369.535177] Lustre: server umount fs11605-OST0000 complete [11373.919158] LustreError: 304135:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11373.959138] LustreError: 304135:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 18 previous similar messages [11383.421335] loop1: detected capacity change from 0 to 204800000 [11385.844548] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11389.340090] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11389.647492] Lustre: fs11605-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11390.952763] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11396.079620] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [11405.531651] loop2: detected capacity change from 0 to 204800000 [11405.781543] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11405.806772] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11405.832168] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11411.257026] loop2: detected capacity change from 0 to 204800000 [11417.361300] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11417.455784] LustreError: fs11605-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11417.475946] LustreError: fs11605-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11417.486412] LustreError: 309084:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -13 [11417.492818] LustreError: 309084:0:(tgt_mount.c:2132:server_put_super()) no obd fs11605-OST0001 [11417.500109] LustreError: 309084:0:(tgt_mount.c:116:server_deregister_mount()) fs11605-OST0001 not registered [11417.532185] Lustre: server umount fs11605-OST0001 complete [11417.536173] LustreError: 309084:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11430.334852] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11430.419364] Lustre: fs11605-OST0001: new disk, initializing [11430.424031] Lustre: srv-fs11605-OST0001: No data found on store. Initialize space. [11430.430495] Lustre: 309634:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11430.495196] Lustre: fs11605-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11438.324328] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [11440.464932] LustreError: 304136:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11440.487522] LustreError: 304136:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [11447.472318] Lustre: server umount fs11605-OST0000 complete [11457.019930] Lustre: server umount fs11605-OST0001 complete [11468.965855] loop1: detected capacity change from 0 to 204800000 [11474.011456] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11474.229882] Lustre: fs11605-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11479.863462] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [11486.880288] loop2: detected capacity change from 0 to 204800000 [11492.572111] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11499.765188] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [11510.025712] Lustre: server umount fs11605-OST0000 complete [11519.348342] Lustre: server umount fs11605-OST0001 complete [11532.260123] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11532.266487] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11532.284297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11534.304913] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11534.308501] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11534.339299] Lustre: Skipped 1 previous similar message [11537.657820] Lustre: server umount lustre-OST0000 complete [11543.009843] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11543.025528] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11543.038063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11549.666393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11549.671014] Lustre: Skipped 2 previous similar messages [11554.786309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11554.789965] Lustre: Skipped 1 previous similar message [11556.319492] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11558.538852] Lustre: server umount lustre-MDT0000 complete [11562.682446] LustreError: 302289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788635919 with bad export cookie 9240742878920915116 [11562.683930] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11562.710108] LustreError: 302289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11563.361841] Lustre: server umount lustre-MDT0001 complete [11574.576712] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [11577.373787] Key type lgssc unregistered [11577.696572] LNet: 315559:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11577.712149] LNetError: 315559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11577.750255] LNet: Removed LNI 192.168.203.123@tcp [11578.846147] Key type .llcrypt unregistered [11578.850127] Key type ._llcrypt unregistered [11610.065368] Key type ._llcrypt registered [11610.068613] Key type .llcrypt registered [11610.220505] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [11627.504100] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [11628.982493] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11629.075939] alg: No test for adler32 (adler32-zlib) [11630.605099] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [11631.000466] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [11632.800746] Key type lgssc registered [11634.192545] Lustre: Echo OBD driver; http://www.lustre.org/ [11677.954351] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11678.019486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11679.359918] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11679.423524] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11679.505690] Lustre: lustre-MDT0000: new disk, initializing [11679.619500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11679.649186] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11683.995112] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [11694.603049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11694.733559] Lustre: 319417:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11694.779221] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11694.782688] Lustre: Skipped 1 previous similar message [11694.853933] Lustre: lustre-MDT0001: new disk, initializing [11694.857140] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [11694.945992] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11694.967369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11694.976588] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11698.841435] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [11708.272568] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11714.059331] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11721.419177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11721.666468] Lustre: lustre-OST0000: new disk, initializing [11721.671234] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11721.676182] Lustre: 321388:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11721.753240] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11728.199837] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [11729.993677] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11730.011715] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11730.130797] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11737.332613] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11739.353813] Lustre: Mounted lustre-client - version 2.17.58_2_g69c2166 [11740.258981] Lustre: Unmounted lustre-client [11749.858596] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11749.875604] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11749.881548] Lustre: Skipped 1 previous similar message [11754.982887] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11754.992971] Lustre: Skipped 1 previous similar message [11755.436632] Lustre: server umount lustre-OST0000 complete [11760.098298] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11760.102844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11760.118969] Lustre: Skipped 2 previous similar messages [11760.125303] Lustre: Skipped 1 previous similar message [11765.221250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11765.228492] Lustre: Skipped 1 previous similar message [11773.407135] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11773.678715] Lustre: server umount lustre-MDT0000 complete [11778.142335] LustreError: 319409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788636135 with bad export cookie 7061545690031300349 [11778.184533] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11778.194084] LustreError: 319409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11778.640762] Lustre: server umount lustre-MDT0001 complete [11790.940366] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [11794.219698] Key type lgssc unregistered [11800.095207] LNet: 1 local NIs in recovery (showing 1): 192.168.203.123@tcp [11800.736992] LNet: 323551:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11800.756376] LNetError: 323551:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11871.471524] LNet: Removed LNI 192.168.203.123@tcp [11872.665849] Key type .llcrypt unregistered [11872.668753] Key type ._llcrypt unregistered [11891.326664] Key type ._llcrypt registered [11891.329909] Key type .llcrypt registered [11891.456620] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [11907.777057] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [11908.876265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11909.009419] alg: No test for adler32 (adler32-zlib) [11910.147190] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [11910.413329] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [11912.095317] Key type lgssc registered [11913.148384] Lustre: Echo OBD driver; http://www.lustre.org/ [11944.766416] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 15:24:59 (1788636299) [11951.503109] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11951.527886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11952.788180] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11952.819305] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11952.879453] Lustre: lustre-MDT0000: new disk, initializing [11953.001472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11953.028596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11956.799681] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [11967.060953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11967.199555] Lustre: 326768:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11967.259192] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11967.266881] Lustre: Skipped 1 previous similar message [11967.364080] Lustre: lustre-MDT0001: new disk, initializing [11967.563686] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11967.643027] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11967.657895] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11972.800903] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [11982.562517] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11988.598452] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11995.774281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11996.126495] Lustre: lustre-OST0000: new disk, initializing [11996.132753] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11996.140210] Lustre: 328738:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11996.235762] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12002.146257] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [12005.472956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12005.483185] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12005.649519] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12011.857472] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12031.460229] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12031.464644] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12031.470875] LustreError: Skipped 1 previous similar message [12031.471606] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12031.498667] Lustre: Skipped 1 previous similar message [12035.063878] Lustre: server umount lustre-OST0000 complete [12041.698843] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12041.704089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12041.738441] Lustre: Skipped 1 previous similar message [12041.755879] Lustre: Skipped 1 previous similar message [12046.896438] Lustre: server umount lustre-MDT0000 complete [12051.941443] LustreError: 326780:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12051.967250] LustreError: 326780:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12052.474586] LustreError: 327563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788636409 with bad export cookie 14995827847775946111 [12052.480262] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12052.502879] LustreError: 327563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12052.974464] Lustre: server umount lustre-MDT0001 complete [12066.280358] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [12069.263750] Key type lgssc unregistered [12069.592609] LNet: 330643:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12069.615860] LNetError: 330643:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12069.645509] LNet: Removed LNI 192.168.203.123@tcp [12070.394581] Key type .llcrypt unregistered [12070.397204] Key type ._llcrypt unregistered [12091.919255] Key type ._llcrypt registered [12091.920994] Key type .llcrypt registered [12091.995367] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [12092.952868] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12093.047421] alg: No test for adler32 (adler32-zlib) [12093.999368] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [12094.213606] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [12095.968309] Key type lgssc registered [12097.730236] Lustre: Echo OBD driver; http://www.lustre.org/ [12108.892740] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12108.950715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12110.444439] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12114.693462] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [12120.549331] LustreError: 331914:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12120.583402] LustreError: 331914:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12124.360697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12124.735939] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12129.019065] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [12137.741334] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12143.690094] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12151.262243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12151.606642] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12158.494793] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [12165.647735] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12179.517279] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12181.090957] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12185.575951] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12185.580992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12185.622763] Lustre: Skipped 2 previous similar messages [12185.634863] Lustre: Skipped 1 previous similar message [12189.032305] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.23@tcp (stopping) [12190.690317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12190.698713] Lustre: Skipped 2 previous similar messages [12194.129544] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.23@tcp (stopping) [12199.255099] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.23@tcp (stopping) [12199.268597] Lustre: Skipped 3 previous similar messages [12199.391158] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12199.605607] Lustre: server umount lustre-MDT0000 complete [12200.942462] LustreError: 332927:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12200.979536] LustreError: 332927:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12203.695339] LustreError: 333758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788636560 with bad export cookie 1608975435290254769 [12203.703898] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12203.706587] LustreError: 333758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12204.364271] LustreError: 331910:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12206.049834] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12208.098552] LustreError: 333149:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12208.099460] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12208.107062] LustreError: 333149:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12208.116179] Lustre: Skipped 2 previous similar messages [12210.208869] Lustre: server umount lustre-MDT0001 complete [12215.012784] Lustre: server umount lustre-OST0000 complete [12225.162150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12230.350749] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [12233.459000] Lustre: 336096:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.123@tcp [12235.882550] Lustre: server umount MGS complete [12245.115331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12245.556950] LustreError: 336669:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12245.583980] LustreError: 336669:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12245.651581] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12249.737947] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [12255.204935] LustreError: 336670:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12255.223307] LustreError: 336670:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12257.977251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12258.489623] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12263.050683] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [12271.212432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12271.533390] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12277.558611] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [12285.766623] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12288.183502] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12294.183069] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12296.220512] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12301.281299] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12301.286441] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12301.303942] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12301.316226] Lustre: Skipped 1 previous similar message [12301.334259] Lustre: Skipped 2 previous similar messages [12307.302456] Lustre: server umount lustre-OST0000 complete [12311.520830] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12317.660815] Lustre: server umount lustre-MDT0000 complete [12321.766192] LustreError: 336669:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12321.788825] LustreError: 336669:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12323.101064] LustreError: 336651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788636680 with bad export cookie 1608975435290255609 [12323.118035] LustreError: 336651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12323.124403] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12323.456455] Lustre: server umount lustre-MDT0001 complete [12337.964514] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [12340.499444] Key type lgssc unregistered [12340.762798] LNet: 340106:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12340.767928] LNetError: 340106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12340.777508] LNet: Removed LNI 192.168.203.123@tcp [12341.437328] Key type .llcrypt unregistered [12341.442642] Key type ._llcrypt unregistered [12358.639113] Key type ._llcrypt registered [12358.641083] Key type .llcrypt registered [12358.735345] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [12373.858608] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [12374.730720] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12374.758676] alg: No test for adler32 (adler32-zlib) [12375.809471] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [12376.037223] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [12377.695194] Key type lgssc registered [12378.810624] Lustre: Echo OBD driver; http://www.lustre.org/ [12416.658588] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12416.692864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12417.932660] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12417.980929] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12418.099596] Lustre: lustre-MDT0000: new disk, initializing [12418.170769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12418.184509] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12422.485951] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [12432.682699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12432.789512] Lustre: 343263:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12432.838901] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12432.844483] Lustre: Skipped 1 previous similar message [12432.916745] Lustre: lustre-MDT0001: new disk, initializing [12432.978385] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12433.010981] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12433.019228] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12437.246859] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [12447.102792] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12452.781486] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12460.325658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12460.665643] Lustre: lustre-OST0000: new disk, initializing [12460.669906] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12460.677969] Lustre: 345232:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12460.781511] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12467.491254] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [12470.362222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12470.378746] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12470.478874] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12477.429438] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12482.362598] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12482.610994] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12487.429936] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12487.745832] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12491.237387] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12491.243100] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12491.247424] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12491.262880] Lustre: Skipped 1 previous similar message [12495.859372] Lustre: server umount lustre-OST0000 complete [12501.474484] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12501.482559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12501.498029] Lustre: Skipped 1 previous similar message [12501.504565] Lustre: Skipped 2 previous similar messages [12506.617905] Lustre: server umount lustre-MDT0000 complete [12510.246372] LustreError: 343257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788636867 with bad export cookie 11832466389385820900 [12510.257200] LustreError: 343257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12510.257399] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12510.615368] Lustre: server umount lustre-MDT0001 complete [12519.810536] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 15:34:34 (1788636874) [12533.855965] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [12543.498889] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [12582.107795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12582.358277] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12582.398379] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12582.493717] Lustre: lustre-MDT0000: new disk, initializing [12582.575872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12582.596926] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12586.978382] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [12597.060049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12597.169953] Lustre: 349991:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12597.236237] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12597.245363] Lustre: Skipped 1 previous similar message [12597.357804] Lustre: lustre-MDT0001: new disk, initializing [12597.446909] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12597.492103] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12597.507461] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12602.032450] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [12611.609502] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12618.204966] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12625.564891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12625.796330] Lustre: lustre-OST0000: new disk, initializing [12625.808855] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12625.818261] Lustre: 351960:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12625.904476] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12632.517960] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [12635.693596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12635.715471] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12635.829958] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12641.623231] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12677.093782] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12677.096270] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12677.110165] LustreError: Skipped 2 previous similar messages [12677.110929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12677.155297] Lustre: Skipped 1 previous similar message [12682.209436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12682.219952] Lustre: Skipped 2 previous similar messages [12687.327122] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12687.339798] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12687.354966] Lustre: Skipped 1 previous similar message [12687.563972] Lustre: server umount lustre-OST0000 complete [12692.451029] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12692.463787] Lustre: Skipped 1 previous similar message [12697.570565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12697.581737] Lustre: Skipped 3 previous similar messages [12705.247230] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12705.472143] Lustre: server umount lustre-MDT0000 complete [12707.810471] LustreError: 350802:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12707.822853] LustreError: 350802:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12708.917403] LustreError: 349984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788637065 with bad export cookie 11832466389385821782 [12708.918091] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12708.934295] LustreError: 349984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12709.314954] Lustre: server umount lustre-MDT0001 complete [12718.176590] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [12720.288894] Key type lgssc unregistered [12720.528160] LNet: 354136:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12720.535954] LNetError: 354136:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12720.555982] LNet: Removed LNI 192.168.203.123@tcp [12721.230813] Key type .llcrypt unregistered [12721.235512] Key type ._llcrypt unregistered [12796.780693] Key type ._llcrypt registered [12796.782487] Key type .llcrypt registered [12796.900397] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [12797.159274] Key type .llcrypt unregistered [12797.162155] Key type ._llcrypt unregistered [12868.675157] Key type ._llcrypt registered [12868.684412] Key type .llcrypt registered [12868.829821] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [12869.338374] Key type .llcrypt unregistered [12869.351078] Key type ._llcrypt unregistered [12900.138092] Key type ._llcrypt registered [12900.143125] Key type .llcrypt registered [12900.275741] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [12900.486400] Key type .llcrypt unregistered [12900.489053] Key type ._llcrypt unregistered [12931.180709] Key type ._llcrypt registered [12931.182907] Key type .llcrypt registered [12931.276524] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [12931.534251] Key type .llcrypt unregistered [12931.536836] Key type ._llcrypt unregistered [13000.369521] Key type ._llcrypt registered [13000.377797] Key type .llcrypt registered [13000.498572] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [13000.773456] Key type .llcrypt unregistered [13000.775391] Key type ._llcrypt unregistered [13029.895861] Key type ._llcrypt registered [13029.898613] Key type .llcrypt registered [13029.957450] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [13030.170292] Key type .llcrypt unregistered [13030.174448] Key type ._llcrypt unregistered [13053.117563] Key type ._llcrypt registered [13053.120439] Key type .llcrypt registered [13053.223674] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [13053.545419] Key type .llcrypt unregistered [13053.548329] Key type ._llcrypt unregistered [13081.472506] Key type ._llcrypt registered [13081.475031] Key type .llcrypt registered [13081.546354] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [13081.786296] Key type .llcrypt unregistered [13081.788288] Key type ._llcrypt unregistered [13109.168466] Key type ._llcrypt registered [13109.170824] Key type .llcrypt registered [13109.275805] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [13109.560325] Key type .llcrypt unregistered [13109.567598] Key type ._llcrypt unregistered [13136.270896] Key type ._llcrypt registered [13136.275994] Key type .llcrypt registered [13136.381705] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [13136.672374] Key type .llcrypt unregistered [13136.680260] Key type ._llcrypt unregistered [13192.808424] Key type ._llcrypt registered [13192.810983] Key type .llcrypt registered [13192.896045] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [13193.131401] Key type .llcrypt unregistered [13193.133510] Key type ._llcrypt unregistered [13220.886336] Key type ._llcrypt registered [13220.889372] Key type .llcrypt registered [13220.963827] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [13221.290351] Key type .llcrypt unregistered [13221.294400] Key type ._llcrypt unregistered [13249.043802] Key type ._llcrypt registered [13249.047516] Key type .llcrypt registered [13249.131892] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [13249.350406] Key type .llcrypt unregistered [13249.355074] Key type ._llcrypt unregistered [13300.114756] Key type ._llcrypt registered [13300.117657] Key type .llcrypt registered [13300.254859] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [13315.424917] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [13316.652039] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13316.674550] alg: No test for adler32 (adler32-zlib) [13317.713852] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [13317.956095] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [13319.615184] Key type lgssc registered [13320.568849] Lustre: Echo OBD driver; http://www.lustre.org/ [13353.051954] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13353.080425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13354.315230] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13354.363313] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13354.435558] Lustre: tst250fs-MDT0000: new disk, initializing [13354.522621] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13354.545346] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13358.912810] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [13367.941409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13368.078617] Lustre: 374318:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13368.166940] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13368.172885] Lustre: Skipped 1 previous similar message [13368.247284] Lustre: tst250fs-MDT0001: new disk, initializing [13368.363138] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13368.392945] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13368.400974] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13373.086344] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [13377.536769] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13383.966060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13384.234700] Lustre: tst250fs-OST0000: new disk, initializing [13384.242268] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13384.251680] Lustre: 375949:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13384.353825] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13385.964767] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13385.979120] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13386.199223] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13390.952215] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [13401.808170] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13402.004539] Lustre: tst250fs-OST0001: new disk, initializing [13402.008543] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13402.013606] Lustre: 376822:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13402.096825] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13408.903319] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [13410.968305] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13410.978664] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13411.087610] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13414.981947] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13417.682975] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [13417.692741] Lustre: Skipped 2 previous similar messages [13436.897565] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13436.906231] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13436.922080] Lustre: Skipped 3 previous similar messages [13436.928234] Lustre: Skipped 2 previous similar messages [13441.403706] Lustre: server umount tst250fs-MDT0000 complete [13444.892049] LustreError: 374310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788637801 with bad export cookie 13373280031326632517 [13444.900696] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13444.907062] LustreError: 374310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13445.333539] Lustre: server umount tst250fs-MDT0001 complete [13459.225943] Lustre: server umount tst250fs-OST0000 complete [13473.448969] Lustre: server umount tst250fs-OST0001 complete [13495.021362] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [13504.280469] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [13538.071532] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 15:51:33 (1788637893) [13539.942718] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13557.297847] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [13568.834942] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [13606.695709] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13360 sec ========= 15:52:41 (1788637961) [13609.294661] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 15:52:43 (1788637963) === [13613.423337] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 15:52:47 (1788637967) === [13654.055898] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [13656.771712] Key type lgssc unregistered [13657.051510] LNet: 385950:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13657.062679] LNetError: 385950:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13658.086194] LNet: Removed LNI 192.168.203.123@tcp [13658.926140] Key type .llcrypt unregistered [13658.928257] Key type ._llcrypt unregistered