[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 477690205 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.003160] x2apic enabled [ 0.004013] Switched APIC routing to physical x2apic. [ 0.005037] kvm-guest: setup PV IPIs [ 0.007786] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008031] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010021] pid_max: default: 32768 minimum: 301 [ 0.012169] LSM: Security Framework initializing [ 0.013085] Yama: becoming mindful. [ 0.014060] SELinux: Initializing. [ 0.015097] *** VALIDATE selinux *** [ 0.024292] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029540] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030196] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032083] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033159] *** VALIDATE tmpfs *** [ 0.035422] *** VALIDATE proc *** [ 0.037034] *** VALIDATE cgroup *** [ 0.038015] *** VALIDATE cgroup2 *** [ 0.040068] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.041192] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.042014] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.043047] Spectre V2 : User space: Vulnerable [ 0.044014] Speculative Store Bypass: Vulnerable [ 0.047742] debug: unmapping init [mem 0xffffffffa7e59000-0xffffffffa7e60fff] [ 0.050167] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.051818] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.052033] ... version: 2 [ 0.053023] ... bit width: 48 [ 0.054019] ... generic registers: 4 [ 0.055021] ... value mask: 0000ffffffffffff [ 0.056019] ... max period: 00007fffffffffff [ 0.057023] ... fixed-purpose events: 3 [ 0.058016] ... event mask: 000000070000000f [ 0.060302] rcu: Hierarchical SRCU implementation. [ 0.062721] smp: Bringing up secondary CPUs ... [ 0.063729] x86: Booting SMP configuration: [ 0.064037] .... node #0, CPUs: #1 #2 #3 [ 0.067784] smp: Brought up 1 node, 4 CPUs [ 0.069016] smpboot: Max logical packages: 1 [ 0.070021] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.257497] node 0 deferred pages initialised in 182ms [ 0.262013] devtmpfs: initialized [ 0.263282] x86/mm: Memory block size: 128MB [ 0.265932] gcov: version magic: 0x41383552 [ 0.269309] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.272084] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.274277] pinctrl core: initialized pinctrl subsystem [ 0.276179] [ 0.276637] ************************************************************* [ 0.278014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.280012] ** ** [ 0.282011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.284011] ** ** [ 0.286012] ** This means that this kernel is built to expose internal ** [ 0.288013] ** IOMMU data structures, which may compromise security on ** [ 0.291012] ** your system. ** [ 0.293010] ** ** [ 0.295011] ** If you see this message and you are not debugging the ** [ 0.297011] ** kernel, report this immediately to your vendor! ** [ 0.299013] ** ** [ 0.301020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.304013] ************************************************************* [ 0.306604] NET: Registered protocol family 16 [ 0.308391] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.310106] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.313088] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.317038] cpuidle: using governor menu [ 0.319791] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.322662] PCI: Using configuration type 1 for base access [ 0.324155] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.334044] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.335022] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.336210] cryptd: max_cpu_qlen set to 1000 [ 0.339292] ACPI: Added _OSI(Module Device) [ 0.340017] ACPI: Added _OSI(Processor Device) [ 0.342053] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.343025] ACPI: Added _OSI(Processor Aggregator Device) [ 0.348432] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.355377] ACPI: Interpreter enabled [ 0.357086] ACPI: PM: (supports S0 S3 S4 S5) [ 0.358017] ACPI: Using IOAPIC for interrupt routing [ 0.360145] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.363475] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.374400] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.377049] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.379033] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.382102] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.388403] acpiphp: Slot [2] registered [ 0.389086] acpiphp: Slot [5] registered [ 0.391103] acpiphp: Slot [6] registered [ 0.392121] acpiphp: Slot [7] registered [ 0.394154] acpiphp: Slot [8] registered [ 0.395143] acpiphp: Slot [9] registered [ 0.397142] acpiphp: Slot [10] registered [ 0.399129] acpiphp: Slot [3] registered [ 0.400127] acpiphp: Slot [4] registered [ 0.402119] acpiphp: Slot [11] registered [ 0.403198] acpiphp: Slot [12] registered [ 0.405112] acpiphp: Slot [13] registered [ 0.406126] acpiphp: Slot [14] registered [ 0.408190] acpiphp: Slot [15] registered [ 0.410166] acpiphp: Slot [16] registered [ 0.411154] acpiphp: Slot [17] registered [ 0.412122] acpiphp: Slot [18] registered [ 0.414115] acpiphp: Slot [19] registered [ 0.415104] acpiphp: Slot [20] registered [ 0.416118] acpiphp: Slot [21] registered [ 0.418115] acpiphp: Slot [22] registered [ 0.419136] acpiphp: Slot [23] registered [ 0.420117] acpiphp: Slot [24] registered [ 0.422107] acpiphp: Slot [25] registered [ 0.423114] acpiphp: Slot [26] registered [ 0.424163] acpiphp: Slot [27] registered [ 0.426129] acpiphp: Slot [28] registered [ 0.427138] acpiphp: Slot [29] registered [ 0.428124] acpiphp: Slot [30] registered [ 0.430111] acpiphp: Slot [31] registered [ 0.431067] PCI host bridge to bus 0000:00 [ 0.432024] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.435032] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.436028] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.439040] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.441027] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.444035] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.445190] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.448080] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.450518] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.461673] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.467579] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.470029] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.473028] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.476026] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.478545] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.481890] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.484043] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.486732] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.492016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.505018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.512820] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.519470] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.528021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.535019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.555035] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.564735] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.572015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.579015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.592024] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.603105] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.610022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.617023] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.634027] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.646951] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.654018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.664015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.676021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.686384] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.694021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.701022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.720022] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.732324] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.753017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.764015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.793019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.813842] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.816494] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.819607] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.821429] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.824291] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.828134] iommu: Default domain type: Passthrough [ 0.830444] SCSI subsystem initialized [ 0.832122] ACPI: bus type USB registered [ 0.834109] usbcore: registered new interface driver usbfs [ 0.835089] usbcore: registered new interface driver hub [ 0.838107] usbcore: registered new device driver usb [ 0.840141] pps_core: LinuxPPS API ver. 1 registered [ 0.842014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.846066] PTP clock support registered [ 0.848204] EDAC MC: Ver: 3.0.0 [ 0.851160] PCI: Using ACPI for IRQ routing [ 0.852918] NetLabel: Initializing [ 0.853013] NetLabel: domain hash size = 128 [ 0.854010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.855138] NetLabel: unlabeled traffic allowed by default [ 0.857115] vgaarb: loaded [ 0.859042] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.861018] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.867427] clocksource: Switched to clocksource kvm-clock [ 0.979668] VFS: Disk quotas dquot_6.6.0 [ 0.981533] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.984446] *** VALIDATE ramfs *** [ 0.985876] *** VALIDATE hugetlbfs *** [ 0.987868] pnp: PnP ACPI init [ 0.990489] pnp: PnP ACPI: found 6 devices [ 1.013422] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.017454] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.019817] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.022689] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.025668] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.028447] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.031563] NET: Registered protocol family 2 [ 1.034597] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.040365] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.043970] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.049367] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.053229] TCP: Hash tables configured (established 65536 bind 65536) [ 1.056676] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.060278] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.063504] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.066743] NET: Registered protocol family 1 [ 1.069553] RPC: Registered named UNIX socket transport module. [ 1.073074] RPC: Registered udp transport module. [ 1.074677] RPC: Registered tcp transport module. [ 1.076021] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.078567] NET: Registered protocol family 44 [ 1.080674] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.082783] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.084751] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.087089] PCI: CLS 0 bytes, default 64 [ 1.089160] Unpacking initramfs... [ 2.491082] debug: unmapping init [mem 0xffff91723cc54000-0xffff91723ffbffff] [ 2.495183] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.497416] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.500379] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.995414] Initialise system trusted keyrings [ 2.997289] Key type blacklist registered [ 2.999047] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.014474] zbud: loaded [ 3.017859] *** VALIDATE nfs *** [ 3.019092] *** VALIDATE nfs4 *** [ 3.020684] pstore: using deflate compression [ 3.024344] Platform Keyring initialized [ 3.126930] NET: Registered protocol family 38 [ 3.128254] Key type asymmetric registered [ 3.129741] Asymmetric key parser 'x509' registered [ 3.131519] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.136419] io scheduler mq-deadline registered [ 3.137953] io scheduler kyber registered [ 3.139265] io scheduler bfq registered [ 3.140844] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.143718] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.146445] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.149311] ACPI: Power Button [PWRF] [ 3.154667] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.161077] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.181112] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.189177] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.209733] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.238860] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.277272] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.292574] Non-volatile memory driver v1.3 [ 3.294287] Linux agpgart interface v0.103 [ 3.335451] virtio_blk virtio1: [vda] 150064 512-byte logical blocks (76.8 MB/73.3 MiB) [ 3.339386] vda: detected capacity change from 0 to 76832768 [ 3.354247] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.357286] vdb: detected capacity change from 0 to 1073741824 [ 3.374649] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.377805] vdc: detected capacity change from 0 to 2621440000 [ 3.393768] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.397046] vdd: detected capacity change from 0 to 2621440000 [ 3.413113] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.415955] vde: detected capacity change from 0 to 4294967296 [ 3.432976] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.439053] vdf: detected capacity change from 0 to 4294967296 [ 3.446495] libphy: Fixed MDIO Bus: probed [ 3.454251] usbcore: registered new interface driver usbserial_generic [ 3.456228] usbserial: USB Serial support registered for generic [ 3.457944] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.461261] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.462514] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.464866] mousedev: PS/2 mouse device common for all mice [ 3.468629] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.469074] rtc_cmos 00:05: RTC can wake from S4 [ 3.476179] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.477119] rtc_cmos 00:05: registered as rtc0 [ 3.484252] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.485218] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.487901] intel_pstate: CPU model not supported [ 3.494444] hid: raw HID events driver (C) Jiri Kosina [ 3.496607] usbcore: registered new interface driver usbhid [ 3.498336] usbhid: USB HID core driver [ 3.499605] drop_monitor: Initializing network drop monitor service [ 3.501279] Initializing XFRM netlink socket [ 3.503193] NET: Registered protocol family 10 [ 3.506411] Segment Routing with IPv6 [ 3.507626] NET: Registered protocol family 17 [ 3.509651] mpls_gso: MPLS GSO support [ 3.514438] RAS: Correctable Errors collector initialized. [ 3.517034] AVX version of gcm_enc/dec engaged. [ 3.518919] AES CTR mode by8 optimization enabled [ 3.591519] sched_clock: Marking stable (3591498239, 0)->(4529907545, -938409306) [ 3.594796] registered taskstats version 1 [ 3.596466] Loading compiled-in X.509 certificates [ 3.598765] zswap: loaded using pool lzo/zbud [ 3.625656] Key type big_key registered [ 3.637106] Key type encrypted registered [ 3.638519] ima: No TPM chip found, activating TPM-bypass! [ 3.640748] ima: Allocated hash algorithm: sha1 [ 3.642510] ima: No architecture policies found [ 3.644077] evm: Initialising EVM extended attributes: [ 3.645814] evm: security.selinux [ 3.646945] evm: security.ima [ 3.648233] evm: security.capability [ 3.649707] evm: HMAC attrs: 0x1 [ 3.653384] rtc_cmos 00:05: setting system clock to 2026-09-07 02:36:01 UTC (1788748561) [ 3.660792] debug: unmapping init [mem 0xffffffffa8e03000-0xffffffffa8ffffff] [ 3.663724] debug: unmapping init [mem 0xffffffffa7b82000-0xffffffffa7e58fff] [ 3.672150] Write protecting the kernel read-only data: 28672k [ 3.675719] debug: unmapping init [mem 0xffffffffa6203000-0xffffffffa63fffff] [ 3.678102] debug: unmapping init [mem 0xffffffffa6b14000-0xffffffffa6bfffff] [ 3.708978] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.716925] systemd[1]: Detected virtualization kvm. [ 3.718610] systemd[1]: Detected architecture x86-64. [ 3.720363] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.744475] systemd[1]: No hostname configured. [ 3.746254] systemd[1]: Set hostname to . [ 3.748308] random: systemd: uninitialized urandom read (16 bytes read) [ 3.750745] systemd[1]: Initializing machine ID from random generator. [ 3.783669] random: ln: uninitialized urandom read (6 bytes read) [ 3.872211] random: systemd: uninitialized urandom read (16 bytes read) [ 3.875500] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.881556] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.886172] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Paths. [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... [ OK ] Reached target Swap. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.522492] device-mapper: uevent: version 1.0.3 [ 4.524662] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.270709] virtio_net virtio0 ens2: renamed from eth0 [ 5.287906] random: fast init done [ 5.353210] scsi host0: ata_piix [ 5.411196] scsi host1: ata_piix [ 5.412877] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.417171] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.528158] dracut-initqueue[588]: RTNETLINK answers: File exists [ 10.160086] random: crng init done [ 10.161975] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.497644] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.573874] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.840340] SELinux: Disabled at runtime. [ 11.901273] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.913014] systemd[1]: Detected virtualization kvm. [ 11.915231] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.441583] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.445098] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.450749] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.454395] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.458969] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.466380] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.470079] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Create list of required st…ce nodes for the current kernel... Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Switch Root. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting POSIX Message Queue File System... [ OK ] Stopped target Initrd Root File System. Mounting Kernel Debug File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 12.570051] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target Paths. Mounting Huge Pages File System... Starting Apply Kernel Variables... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.901944] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.174810] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.187026] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.289595] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.300100] EDAC sbridge: Ver: 1.1.2 [ 14.922562] Key type dns_resolver registered [ 15.247691] NFS: Registering the id_resolver key type [ 15.250195] Key type id_resolver registered [ 15.251780] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg608-server login: [ 70.611278] libcfs: loading out-of-tree module taints kernel. [ 70.652886] Key type ._llcrypt registered [ 70.655169] Key type .llcrypt registered [ 70.759457] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 87.487475] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 88.786814] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 88.817679] alg: No test for adler32 (adler32-zlib) [ 90.169519] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 91.024767] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 92.872281] Key type lgssc registered [ 94.468848] Lustre: Echo OBD driver; http://www.lustre.org/ [ 111.923897] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 148.528119] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 161.900477] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 161.955280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 163.211346] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 163.242329] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 163.361952] Lustre: lustre-MDT0000: new disk, initializing [ 163.459376] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 163.477253] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 168.676568] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 180.906538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 181.032879] Lustre: 6514:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 181.076031] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 181.082251] Lustre: Skipped 1 previous similar message [ 181.195114] Lustre: lustre-MDT0001: new disk, initializing [ 181.278168] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 181.320592] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 181.327753] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 185.083727] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 189.568659] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 199.143908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 199.352695] Lustre: lustre-OST0000: new disk, initializing [ 199.355927] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 199.360439] Lustre: 8454:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 199.425488] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 199.852596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 199.860895] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 199.969240] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 205.833616] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 218.300594] hrtimer: interrupt took 3939468 ns [ 219.565744] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 219.683772] Lustre: lustre-OST0001: new disk, initializing [ 219.686795] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 219.691735] Lustre: 9525:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 219.761793] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 225.980671] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 229.376701] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 229.392359] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 229.439215] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 237.362730] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 242.376419] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 247.625885] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing check_logdir /tmp/testlogs/ [ 251.600828] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing yml_node [ 254.743774] Lustre: DEBUG MARKER: Client: 2.17.58.41 [ 256.855809] Lustre: DEBUG MARKER: MDS: 2.17.58.41 [ 259.219588] Lustre: DEBUG MARKER: OSS: 2.17.58.41 [ 260.477678] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Sep 6 22:40:16 EDT 2026 [ 274.235697] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 275.294710] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 290.784650] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 290.787435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 290.798837] Lustre: Skipped 1 previous similar message [ 290.811137] Lustre: Skipped 3 previous similar messages [ 295.063670] Lustre: server umount lustre-MDT0000 complete [ 297.895946] LustreError: 6506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788748855 with bad export cookie 14170121771814934344 [ 297.902131] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 297.914110] LustreError: 6506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 301.027068] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 301.027893] LustreError: 6519:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 301.028061] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 301.043350] Lustre: Skipped 3 previous similar messages [ 301.065215] LustreError: 6519:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 6 previous similar messages [ 302.048871] LustreError: 7796:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 302.049222] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 302.058273] Lustre: Skipped 2 previous similar messages [ 307.169533] LustreError: 6520:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 307.172102] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 307.189278] LustreError: 6520:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 307.206162] Lustre: Skipped 1 previous similar message [ 312.289662] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 312.290336] LustreError: 6519:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 312.290397] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 312.308722] LustreError: 6519:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 312.510379] Lustre: server umount lustre-MDT0001 complete [ 315.182620] Lustre: server umount lustre-OST0000 complete [ 317.964530] Lustre: server umount lustre-OST0001 complete [ 324.079748] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 330.529429] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 356.118477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 356.282315] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 356.299934] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 356.369759] Lustre: lustre-MDT0000: new disk, initializing [ 356.446113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 356.457202] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 359.807372] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 367.263506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 367.373276] Lustre: 15073:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 367.424207] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 367.429434] Lustre: Skipped 1 previous similar message [ 367.491610] Lustre: lustre-MDT0001: new disk, initializing [ 367.579623] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 367.607777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 367.612638] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 370.131800] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 376.105962] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 380.094923] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 384.500638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 384.729797] Lustre: lustre-OST0000: new disk, initializing [ 384.732971] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 384.739639] Lustre: 17040:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 384.825036] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 386.522718] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 386.533467] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 386.638063] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 389.618364] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 396.613172] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 400.120144] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 400.310472] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 403.511311] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 403.677909] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 407.013136] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 407.029484] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 407.039092] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 407.039949] Lustre: Skipped 2 previous similar messages [ 419.296112] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 419.413946] Lustre: server umount lustre-OST0000 complete [ 422.374314] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 422.385882] Lustre: Skipped 2 previous similar messages [ 427.490164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 427.494899] Lustre: Skipped 6 previous similar messages [ 428.249391] Lustre: server umount lustre-MDT0000 complete [ 430.846943] LustreError: 15063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788748988 with bad export cookie 14170121771814935877 [ 430.848406] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 430.852619] LustreError: 15063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 431.135957] Lustre: server umount lustre-MDT0001 complete [ 434.574801] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 22:43:11 (1788748991) [ 446.377877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 446.677658] LustreError: 19913:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 446.731927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 449.585496] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 455.428715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 458.863410] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 463.700625] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 467.036391] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 468.336361] Lustre: Failing over lustre-MDT0000 [ 468.537695] Lustre: server umount lustre-MDT0000 complete [ 471.011529] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 471.013639] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 471.020372] LustreError: 19914:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 471.027159] Lustre: Skipped 1 previous similar message [ 471.041518] LustreError: 19914:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 483.949594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 484.027796] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 487.030932] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 489.445834] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 489.450870] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 489.466099] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 492.882799] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 498.658894] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 498.664693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 498.670561] Lustre: Skipped 1 previous similar message [ 498.680712] Lustre: Skipped 1 previous similar message [ 501.878862] Lustre: server umount lustre-MDT0000 complete [ 504.396371] LustreError: 19895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788749062 with bad export cookie 14170121771814937291 [ 504.398990] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 504.402925] LustreError: 19895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 504.608380] Lustre: server umount lustre-MDT0001 complete [ 510.183412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 510.450598] LustreError: 23246:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 510.465748] LustreError: 23246:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 4 previous similar messages [ 513.624439] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 519.755933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 519.998386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 520.005283] Lustre: Skipped 3 previous similar messages [ 522.849840] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 527.554165] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 531.111514] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 532.438882] Lustre: Failing over lustre-MDT0000 [ 532.683911] Lustre: server umount lustre-MDT0000 complete [ 535.521896] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 535.523288] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 535.538375] Lustre: Skipped 1 previous similar message [ 545.762224] LustreError: 23245:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 545.772413] LustreError: 23245:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 6 previous similar messages [ 548.857384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 548.944906] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 552.027980] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 554.474603] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 554.477766] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 554.495824] Lustre: Skipped 1 previous similar message [ 554.506886] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 558.995508] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 563.683141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 563.687449] Lustre: Skipped 2 previous similar messages [ 568.029465] Lustre: server umount lustre-MDT0000 complete [ 570.666943] LustreError: 23227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788749128 with bad export cookie 14170121771814938229 [ 570.673176] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 570.683133] LustreError: 23227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 570.928473] Lustre: server umount lustre-MDT0001 complete [ 577.191929] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 22:45:33 (1788749133) [ 587.869976] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 594.666945] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 619.760186] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 626.926821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 627.101769] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 627.120193] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 627.174461] Lustre: lustre-MDT0000: new disk, initializing [ 627.237399] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 630.102874] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 637.445968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 637.545632] Lustre: 29607:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 637.749297] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 641.170533] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 647.698488] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 651.563438] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 656.038839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 656.210215] Lustre: 31574:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 656.293428] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 656.299324] Lustre: Skipped 3 previous similar messages [ 658.088170] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 658.115785] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 658.158231] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 658.169348] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 660.406607] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 667.130356] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 672.668747] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 687.079317] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 687.100740] Lustre: Skipped 3 previous similar messages [ 691.685177] Lustre: server umount lustre-OST0000 complete [ 697.315294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 697.320553] Lustre: Skipped 6 previous similar messages [ 700.545344] Lustre: server umount lustre-MDT0000 complete [ 702.433895] LustreError: 29620:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 702.444812] LustreError: 29620:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 702.524500] LustreError: 29599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788749260 with bad export cookie 14170121771814938635 [ 702.526323] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 702.532867] LustreError: 29599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 702.775482] Lustre: server umount lustre-MDT0001 complete [ 708.323204] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 709.959418] Key type lgssc unregistered [ 710.134988] LNet: 33549:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 710.143460] LNetError: 33549:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 711.211439] LNet: Removed LNI 192.168.206.108@tcp [ 711.748148] Key type .llcrypt unregistered [ 711.749823] Key type ._llcrypt unregistered [ 724.726889] Key type ._llcrypt registered [ 724.728427] Key type .llcrypt registered [ 724.778305] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 725.218272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 725.230677] alg: No test for adler32 (adler32-zlib) [ 726.133811] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 726.282038] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 727.936607] Key type lgssc registered [ 728.558408] Lustre: Echo OBD driver; http://www.lustre.org/ [ 732.836174] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 732.847300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 734.113722] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 736.176916] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 740.558910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 740.777548] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 743.099652] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 747.065821] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 749.853442] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 753.276758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 753.466656] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 756.187265] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 759.494517] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 765.741270] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 770.092900] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 22:48:46 (1788749326) [ 771.028086] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 771.881185] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 774.742598] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 22:48:51 (1788749331) [ 778.714884] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 22:48:55 (1788749335) [ 779.229291] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 783.514121] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 783.519315] Lustre: Skipped 2 previous similar messages [ 784.490209] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 22:49:01 (1788749341) [ 788.954461] Lustre: Setting parameter lustre-OST0000-osc.osc.max_pages_per_rpc=1024 in log lustre-client [ 788.958566] Lustre: Skipped 4 previous similar messages [ 796.860381] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 22:49:13 (1788749353) [ 798.140388] Lustre: *** cfs_fail_loc=131b, val=2*** [ 799.674518] Lustre: *** cfs_fail_loc=131b, val=2*** [ 803.247862] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 22:49:20 (1788749360) [ 804.065818] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 804.071581] Lustre: Skipped 9 previous similar messages [ 810.347950] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 22:49:27 (1788749367) [ 817.124231] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 817.126463] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 817.128439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 817.144227] Lustre: Skipped 1 previous similar message [ 822.241653] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 822.244714] Lustre: Skipped 2 previous similar messages [ 822.884229] Lustre: server umount lustre-OST0000 complete [ 827.362784] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 827.363842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 827.367108] Lustre: Skipped 1 previous similar message [ 827.371182] Lustre: Skipped 1 previous similar message [ 830.716965] Lustre: server umount lustre-MDT0000 complete [ 832.482524] LustreError: 34937:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 832.497273] LustreError: 34937:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 832.581397] LustreError: 34916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788749390 with bad export cookie 2366207847639614417 [ 832.584809] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 832.590711] LustreError: 34916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 832.742461] Lustre: server umount lustre-MDT0001 complete [ 837.772327] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 839.049828] Key type lgssc unregistered [ 839.198541] LNet: 42774:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 839.201399] LNetError: 42774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 839.209382] LNet: Removed LNI 192.168.206.108@tcp [ 839.550518] Key type .llcrypt unregistered [ 839.551952] Key type ._llcrypt unregistered [ 847.449867] Key type ._llcrypt registered [ 847.451407] Key type .llcrypt registered [ 847.499084] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 855.537863] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 856.042705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 856.051561] alg: No test for adler32 (adler32-zlib) [ 856.970259] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 857.079385] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 858.696197] Key type lgssc registered [ 859.170221] Lustre: Echo OBD driver; http://www.lustre.org/ [ 875.005059] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 875.015792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 876.128979] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 876.154852] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 876.196686] Lustre: lustre-MDT0000: new disk, initializing [ 876.234736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 876.245595] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 878.114895] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 883.551748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 883.606977] Lustre: 45897:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 883.638307] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 883.640996] Lustre: Skipped 1 previous similar message [ 883.683057] Lustre: lustre-MDT0001: new disk, initializing [ 883.725650] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 883.746075] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 883.755337] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 885.513206] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 890.600517] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 893.197456] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 896.777401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 896.916670] Lustre: lustre-OST0000: new disk, initializing [ 896.921345] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 896.929158] Lustre: 47863:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 896.984615] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 899.458093] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 904.100253] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 904.709913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 904.718066] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 904.770038] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 906.075404] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 906.171802] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 908.086001] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 908.195726] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 909.795359] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 909.796253] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 909.802918] LustreError: Skipped 1 previous similar message [ 909.803882] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 909.807055] Lustre: Skipped 1 previous similar message [ 909.821633] Lustre: Skipped 1 previous similar message [ 911.300888] Lustre: server umount lustre-OST0000 complete [ 914.401655] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 914.405953] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 914.416497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 919.437964] Lustre: server umount lustre-MDT0000 complete [ 921.180781] LustreError: 47862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788749479 with bad export cookie 7043146433477255976 [ 921.186344] LustreError: 47862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 921.187103] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 921.340539] Lustre: server umount lustre-MDT0001 complete [ 926.154104] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 22:51:22 (1788749482) [ 929.555062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 929.780834] LustreError: 50321:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 929.788628] LustreError: 50321:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 929.812390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 931.801452] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 934.881405] LustreError: 50322:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 935.836170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 936.019850] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 937.943946] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 941.377823] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 944.041351] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 947.171928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 947.328448] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 950.159884] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 954.063477] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 966.476147] LustreError: 52870:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 966.965208] Lustre: Modifying parameter general.timeout=1 in log params [ 968.445140] Lustre: Modifying parameter general.timeout=4 in log params [ 968.447689] Lustre: Skipped 2 previous similar messages [ 970.651509] Lustre: Modifying parameter general.timeout=9 in log params [ 970.654602] Lustre: Skipped 4 previous similar messages [ 974.700026] Lustre: Modifying parameter general.timeout=19 in log params [ 974.702273] Lustre: Skipped 9 previous similar messages [ 982.777544] Lustre: Modifying parameter general.timeout=38 in log params [ 982.781972] Lustre: Skipped 18 previous similar messages [ 999.200581] Lustre: Modifying parameter general.timeout=75 in log params [ 999.203995] Lustre: Skipped 36 previous similar messages [ 1028.340809] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 22:53:04 (1788749584) [ 1030.257499] Lustre: 59779:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1031.444866] Lustre: 59878:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1031.909823] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 in log params [ 1031.915111] Lustre: Skipped 58 previous similar messages [ 1033.446716] Lustre: 60074:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1033.456577] Lustre: 60074:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1051.025837] Lustre: 61645:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1051.031609] Lustre: 61645:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1051.042276] LustreError: 61645:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1053.198816] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 22:53:29 (1788749609) [ 1061.346618] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1061.347871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1061.357079] Lustre: Skipped 3 previous similar messages [ 1061.360025] Lustre: Skipped 2 previous similar messages [ 1065.451600] Lustre: server umount lustre-MDT0000 complete [ 1066.467817] LustreError: 50322:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1066.475719] LustreError: 50322:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 1067.151195] LustreError: 52161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788749624 with bad export cookie 7043146433477256858 [ 1067.154144] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1067.157384] LustreError: 52161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1067.330302] Lustre: server umount lustre-MDT0001 complete [ 1079.586772] Lustre: server umount lustre-OST0000 complete [ 1091.802627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1091.956715] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1092.092222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1093.907705] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1097.704111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1097.765243] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1097.779902] Lustre: 64170:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1099.800682] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1103.244106] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1105.840149] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1108.917562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1109.013278] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1111.943899] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1115.678053] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1121.830721] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1121.836943] Lustre: Skipped 19 previous similar messages [ 1121.861803] Lustre: 66736:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1127.395147] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1127.395425] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1127.400186] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1127.404097] Lustre: Skipped 1 previous similar message [ 1130.589029] Lustre: server umount lustre-OST0000 complete [ 1132.513423] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1132.514632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1132.519359] Lustre: Skipped 1 previous similar message [ 1132.523532] Lustre: Skipped 1 previous similar message [ 1138.343476] Lustre: server umount lustre-MDT0000 complete [ 1139.815241] LustreError: 64161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788749697 with bad export cookie 7043146433477259364 [ 1139.819280] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1139.820097] LustreError: 64161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1139.950690] Lustre: server umount lustre-MDT0001 complete [ 1144.379821] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 1145.604787] Key type lgssc unregistered [ 1145.749858] LNet: 67943:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1145.755538] LNetError: 67943:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1145.768360] LNet: Removed LNI 192.168.206.108@tcp [ 1146.142957] Key type .llcrypt unregistered [ 1146.144689] Key type ._llcrypt unregistered [ 1157.915716] Key type ._llcrypt registered [ 1157.917385] Key type .llcrypt registered [ 1157.968129] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 1158.412129] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1158.424751] alg: No test for adler32 (adler32-zlib) [ 1159.315724] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 1159.444029] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 1161.048198] Key type lgssc registered [ 1161.573748] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1165.180434] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1165.189758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1166.416090] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1168.451314] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1173.024799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1173.337107] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1175.722199] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1179.631424] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1182.630933] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1186.276490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1186.579547] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1189.251051] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1192.632658] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1200.863515] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1209.317125] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1209.317229] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1209.323801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1209.339423] Lustre: Skipped 2 previous similar messages [ 1210.791463] Lustre: server umount lustre-MDT0000 complete [ 1212.535943] LustreError: 71179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788749770 with bad export cookie 8810843383599330911 [ 1212.536305] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1212.542498] LustreError: 71179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1212.552963] LustreError: 69321:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 1212.693964] Lustre: server umount lustre-MDT0001 complete [ 1243.163861] Lustre: server umount lustre-OST0000 complete [ 1256.140053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1256.292791] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1256.314916] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1256.450073] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1258.424669] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1262.854606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1262.920577] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1262.943907] Lustre: 74197:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1263.226333] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1265.389215] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1269.746536] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1272.687156] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1276.035393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1276.132690] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1276.328377] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1279.498768] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1282.874618] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1288.714476] Lustre: 76771:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1289.519718] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1294.819623] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1294.819718] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1294.820658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1294.820663] Lustre: Skipped 2 previous similar messages [ 1294.824019] LustreError: Skipped 1 previous similar message [ 1294.835047] Lustre: Skipped 1 previous similar message [ 1297.379730] Lustre: server umount lustre-OST0000 complete [ 1299.937916] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1299.942338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1299.947507] Lustre: Skipped 1 previous similar message [ 1299.955645] Lustre: Skipped 2 previous similar messages [ 1305.057174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1305.066144] Lustre: Skipped 1 previous similar message [ 1305.449448] Lustre: server umount lustre-MDT0000 complete [ 1307.763954] LustreError: 74187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788749865 with bad export cookie 8810843383599331723 [ 1307.767893] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1307.776360] LustreError: 74187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1308.008543] Lustre: server umount lustre-MDT0001 complete [ 1314.016217] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 1315.432437] Key type lgssc unregistered [ 1315.589799] LNet: 77978:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1315.595382] LNetError: 77978:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1315.608212] LNet: Removed LNI 192.168.206.108@tcp [ 1316.065345] Key type .llcrypt unregistered [ 1316.068198] Key type ._llcrypt unregistered [ 1328.727502] Key type ._llcrypt registered [ 1328.729059] Key type .llcrypt registered [ 1328.777175] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 1329.249161] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1329.261890] alg: No test for adler32 (adler32-zlib) [ 1330.137315] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 1330.247358] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 1331.840192] Key type lgssc registered [ 1332.311316] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1335.824480] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1335.834760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1337.038082] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1338.579951] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1341.835978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1342.071434] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1343.525935] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1346.131714] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1348.189648] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1350.765193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1350.980175] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1353.020466] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1355.742667] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1364.754055] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1365.523097] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1365.526104] Lustre: Skipped 1 previous similar message [ 1366.679855] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1366.682490] Lustre: Skipped 2 previous similar messages [ 1369.038500] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1369.040629] Lustre: Skipped 5 previous similar messages [ 1373.318903] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1373.321304] Lustre: Skipped 10 previous similar messages [ 1381.621993] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1381.625154] Lustre: Skipped 21 previous similar messages [ 1397.940182] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1397.944433] Lustre: Skipped 43 previous similar messages [ 1405.264836] Lustre: 87427:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1406.241244] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 22:59:23 (1788749963) [ 1406.858613] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1407.566544] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 22:59:24 (1788749964) [ 1411.556374] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1411.557396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1411.562666] Lustre: Skipped 1 previous similar message [ 1411.567881] Lustre: Skipped 1 previous similar message [ 1414.743830] Lustre: server umount lustre-OST0000 complete [ 1416.674498] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1416.677137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1416.679836] Lustre: Skipped 1 previous similar message [ 1421.794533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1421.797019] Lustre: Skipped 2 previous similar messages [ 1422.560202] Lustre: server umount lustre-MDT0000 complete [ 1424.200343] LustreError: 79344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788749982 with bad export cookie 5563143589798811243 [ 1424.204506] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1424.205535] LustreError: 79344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1424.346516] Lustre: server umount lustre-MDT0001 complete [ 1429.068514] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 1430.306542] Key type lgssc unregistered [ 1430.464502] LNet: 88672:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1430.467181] LNetError: 88672:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1431.526548] LNet: Removed LNI 192.168.206.108@tcp [ 1431.932913] Key type .llcrypt unregistered [ 1431.934665] Key type ._llcrypt unregistered [ 1433.630498] Key type ._llcrypt registered [ 1433.631906] Key type .llcrypt registered [ 1433.676484] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1435.373457] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules [ 1435.694956] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1435.705079] alg: No test for adler32 (adler32-zlib) [ 1436.586503] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 1436.702407] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 1438.304186] Key type lgssc registered [ 1438.778345] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1439.268154] LustreError: 89837:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 1447.871296] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 1451.441051] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1460.563425] LustreError: 89837:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1460.577079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1461.778608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1463.660319] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1467.854307] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 23:00:24 (1788750024) [ 1470.884221] LustreError: 90892:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1470.891242] LustreError: 90892:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1470.895944] LustreError: 90892:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 1471.064654] Lustre: server umount lustre-MDT0000 complete [ 1483.078861] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 1486.190954] Key type lgssc unregistered [ 1486.544081] LNet: 92383:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1486.567621] LNetError: 92383:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1486.583426] LNet: Removed LNI 192.168.206.108@tcp [ 1487.749501] Key type .llcrypt unregistered [ 1487.752609] Key type ._llcrypt unregistered [ 1509.728584] Key type ._llcrypt registered [ 1509.733651] Key type .llcrypt registered [ 1509.890535] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 1511.204919] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1511.240286] alg: No test for adler32 (adler32-zlib) [ 1512.576563] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 1512.935218] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 1514.640358] Key type lgssc registered [ 1515.761399] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1524.318191] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1524.342615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1526.038630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1531.260442] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1535.474746] LustreError: 93612:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1535.496762] LustreError: 93612:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 1540.584166] LustreError: 93611:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1542.034784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1543.216711] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1550.190503] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1560.578503] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1566.942846] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1573.700668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1574.198139] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1578.834750] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1584.935578] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1634.294287] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 23:03:10 (1788750190) [ 1636.164855] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 1637.770313] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 23:03:13 (1788750193) [ 1641.954938] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1641.958650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1641.962536] Lustre: Skipped 2 previous similar messages [ 1641.969358] Lustre: Skipped 2 previous similar messages [ 1647.079870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1647.085417] Lustre: Skipped 2 previous similar messages [ 1647.404617] Lustre: server umount lustre-MDT0000 complete [ 1650.672519] LustreError: 93590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788750208 with bad export cookie 11408100974430605651 [ 1650.677639] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1650.685726] LustreError: 93590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1650.994372] Lustre: server umount lustre-MDT0001 complete [ 1663.471978] Lustre: server umount lustre-OST0000 complete [ 1672.441724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1672.694452] LustreError: 97905:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1672.824638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1676.446408] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1678.307304] LustreError: 97906:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1682.404244] LustreError: 97905:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1682.726291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1683.331935] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1686.832608] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1692.557292] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1697.148418] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1707.623593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1707.851392] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1707.856502] LustreError: 97895:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1707.881282] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1707.887838] LustreError: 99987:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 1707.895286] LustreError: 99987:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 1707.901704] LustreError: 99987:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 1707.983971] Lustre: server umount lustre-OST0000 complete [ 1707.986460] LustreError: 99987:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1716.140272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1716.309794] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1716.316995] LustreError: 97893:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1716.335519] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1716.342638] LustreError: 100453:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 1716.347802] LustreError: 100453:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 1716.352771] LustreError: 100453:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 1716.403827] Lustre: server umount lustre-OST0000 complete [ 1716.410625] LustreError: 100453:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1725.445290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1725.599729] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1725.603650] Lustre: Found index 0 for lustre-OST0000, updating log [ 1725.612023] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 1725.685488] Lustre: lustre-OST0000: new disk, initializing [ 1725.693683] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1725.702097] Lustre: 100986:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1725.949843] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1732.256423] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1735.546559] Lustre: Failing over lustre-OST0000 [ 1735.800041] Lustre: server umount lustre-OST0000 complete [ 1743.334618] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1743.337997] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1743.351365] Lustre: Skipped 1 previous similar message [ 1745.940167] Lustre: server umount lustre-MDT0000 complete [ 1748.455692] LustreError: 97905:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1748.485273] LustreError: 97905:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 1749.443458] LustreError: 97885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788750307 with bad export cookie 11408100974430608402 [ 1749.445885] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1749.463207] LustreError: 97885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1759.841908] Lustre: server umount lustre-MDT0001 complete [ 1767.447549] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 23:05:23 (1788750323) [ 1783.739600] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 1792.833258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1793.181755] LustreError: 103335:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1793.249371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1796.923824] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1806.478127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1811.075896] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1813.870224] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 1820.052214] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1820.819692] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 1826.846598] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1832.039166] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1832.070386] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1832.070735] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1835.435270] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1835.504508] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 1835.599407] Lustre: lustre-OST0001: new disk, initializing [ 1835.603467] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1835.609337] Lustre: 105655:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1835.885408] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1835.891049] Lustre: Skipped 2 previous similar messages [ 1841.238321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1841.247545] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1841.369675] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1842.000576] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1849.733386] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1857.107583] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1862.118205] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1862.124649] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1862.142501] Lustre: Skipped 2 previous similar messages [ 1862.158057] Lustre: Skipped 3 previous similar messages [ 1867.241401] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1867.249871] Lustre: Skipped 1 previous similar message [ 1868.335616] Lustre: server umount lustre-MDT0001 complete [ 1870.415357] LustreError: 104852:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1870.451718] LustreError: 104852:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 1878.131537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1878.249224] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1878.253435] Lustre: Found index 1 for lustre-MDT0001, updating log [ 1878.316790] Lustre: 103323:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1880.370638] LustreError: 107239:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 1880.375205] LustreError: 107239:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 1880.396095] LustreError: 103327:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 1880.556475] LustreError: 107245:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.108@tcp: cfg command failed: rc = -17 [ 1880.564697] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 1880.564697] [ 1882.985859] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1884.150994] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1893.769582] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 23:07:29 (1788750449) [ 1912.569925] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 1931.363191] Lustre: 109270:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1954.120591] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1958.071503] Lustre: 110404:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2047.462500] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2047.473701] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2047.491505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2047.495304] Lustre: Skipped 2 previous similar messages [ 2052.940599] Lustre: server umount lustre-MDT0000 complete [ 2053.091691] LustreError: 103331:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2053.106656] LustreError: 103331:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 9 previous similar messages [ 2056.366894] LustreError: 105654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788750614 with bad export cookie 11408100974430609403 [ 2056.370400] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2056.384764] LustreError: 105654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2056.813637] Lustre: server umount lustre-MDT0001 complete [ 2072.406961] Lustre: server umount lustre-OST0000 complete [ 2074.407856] Lustre: 92921:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788750616/real 1788750616] req@ffff917181944000 x1875640393756416/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788750632 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2074.453478] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2074.469432] Lustre: Skipped 3 previous similar messages [ 2076.368764] Lustre: server umount lustre-OST0001 complete [ 2082.335661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2092.984059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2103.130521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2115.154566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2136.214456] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 2146.836298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2146.871694] Lustre: lustre-MDT0000: reset Object Index mappings [ 2147.363883] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2147.370641] Lustre: Skipped 1 previous similar message [ 2151.254462] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2161.722517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2161.754462] Lustre: lustre-MDT0001: reset Object Index mappings [ 2166.952859] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2170.181782] Lustre: 115501:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2177.181570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2185.377445] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2186.234528] LustreError: 115853:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2186.265365] LustreError: 115853:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 2186.265712] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:226 to 0x280000400:257) [ 2192.387421] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 2194.418450] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2200.061921] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 2200.069969] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 2201.660944] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2210.179147] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2214.743481] Lustre: 117379:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2266.329685] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 23:13:42 (1788750822) [ 2292.199172] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2292.202606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2292.211541] Lustre: Skipped 2 previous similar messages [ 2292.215199] Lustre: Skipped 5 previous similar messages [ 2298.138990] Lustre: server umount lustre-MDT0000 complete [ 2301.197887] LustreError: 114336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788750859 with bad export cookie 11408100974430656275 [ 2301.201949] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2301.203059] LustreError: 114336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2307.678683] Lustre: server umount lustre-MDT0001 complete [ 2317.394377] Lustre: server umount lustre-OST0000 complete [ 2326.857928] Lustre: server umount lustre-OST0001 complete [ 2333.906896] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 2343.564506] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 2380.865851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2381.142825] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2381.192147] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2381.306856] Lustre: lustre-MDT0000: new disk, initializing [ 2381.396857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2381.403043] Lustre: Skipped 3 previous similar messages [ 2381.420342] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2385.604940] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2392.882071] Lustre: server umount lustre-MDT0000 complete [ 2402.002689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2402.210235] Lustre: Found index 0 for lustre-MDT0000, updating log [ 2402.224914] Lustre: 121661:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2402.246857] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 2407.115735] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2411.669854] Lustre: server umount lustre-MDT0000 complete [ 2419.408545] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 23:16:15 (1788750975) [ 2421.418666] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 2423.337513] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 23:16:19 (1788750979) [ 2440.503236] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 2450.200368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2454.548148] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2462.335945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2462.465545] Lustre: 123714:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2462.535776] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2462.539853] Lustre: Skipped 1 previous similar message [ 2462.625187] Lustre: lustre-MDT0001: new disk, initializing [ 2462.748344] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2462.755180] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2467.072488] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2471.526263] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2477.669103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2477.886925] Lustre: lustre-OST0000: new disk, initializing [ 2477.890601] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2477.898398] Lustre: 125288:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2479.333116] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2479.339710] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2479.425925] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2482.874271] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2491.097945] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2491.218683] Lustre: lustre-OST0001: new disk, initializing [ 2491.229685] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2491.236632] Lustre: 126159:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2493.186386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2493.198413] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2493.277679] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2495.918407] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2503.124756] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2505.845171] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2831.397298] Lustre: 132567:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 2831.403416] Lustre: 132567:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 2831.409297] Lustre: 132567:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 2831.416270] Lustre: 132567:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 2831.422704] Lustre: 132567:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 2831.430414] Lustre: 132567:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3068.557886] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 23:27:04 (1788751624) [ 3087.333790] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3087.345377] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3087.345833] Lustre: Skipped 4 previous similar messages [ 3087.345983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3087.345990] Lustre: Skipped 9 previous similar messages [ 3088.841475] Lustre: server umount lustre-MDT0000 complete [ 3091.574132] LustreError: 123707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788751649 with bad export cookie 11408100974430724644 [ 3091.580333] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3091.584047] LustreError: 123707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3092.452511] LustreError: 123721:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3092.453062] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3092.456695] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3092.456703] Lustre: Skipped 3 previous similar messages [ 3092.469764] LustreError: 123721:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 8 previous similar messages [ 3092.481780] Lustre: Skipped 2 previous similar messages [ 3097.955864] Lustre: server umount lustre-MDT0001 complete [ 3107.338348] Lustre: server umount lustre-OST0000 complete [ 3116.900710] Lustre: server umount lustre-OST0001 complete [ 3122.593140] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 3130.622579] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 3159.173901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3159.359861] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3159.387089] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3159.446390] Lustre: lustre-MDT0000: new disk, initializing [ 3159.523139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3159.529236] Lustre: Skipped 5 previous similar messages [ 3159.542247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3163.150967] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3171.858937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3172.010807] Lustre: 135613:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3172.080462] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3172.085720] Lustre: Skipped 1 previous similar message [ 3172.160570] Lustre: lustre-MDT0001: new disk, initializing [ 3172.288723] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3172.309665] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3176.487167] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3184.855957] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3189.979031] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3195.990409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3196.288697] Lustre: lustre-OST0000: new disk, initializing [ 3196.292550] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3196.300524] Lustre: 137582:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3197.952773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3197.968766] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3198.081529] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3201.847277] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3210.172466] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3219.427980] Lustre: lustre-MDD0000: changelog on [ 3221.776752] Lustre: lustre-MDD0001: changelog on [ 4579.545676] Lustre: lustre-MDD0001: changelog off [ 4580.529266] Lustre: 135619:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 4581.004779] Lustre: lustre-MDD0000: changelog off [ 4585.441585] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4585.445372] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4585.451527] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4585.454210] Lustre: Skipped 3 previous similar messages [ 4585.953289] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4585.956746] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4588.516691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4588.521765] Lustre: Skipped 2 previous similar messages [ 4589.552107] Lustre: server umount lustre-OST0000 complete [ 4593.634355] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4593.635933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4593.641792] Lustre: Skipped 1 previous similar message [ 4593.648291] Lustre: Skipped 1 previous similar message [ 4597.534084] Lustre: server umount lustre-MDT0000 complete [ 4598.753638] LustreError: 144147:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4598.760697] LustreError: 144147:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 4599.146222] LustreError: 135606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753156 with bad export cookie 11408100974431542559 [ 4599.149063] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4599.151026] LustreError: 135606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4599.305090] Lustre: server umount lustre-MDT0001 complete [ 4604.040395] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 4605.330699] Key type lgssc unregistered [ 4605.497665] LNet: 146628:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4605.500962] LNetError: 146628:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4605.513110] LNet: Removed LNI 192.168.206.108@tcp [ 4605.925272] Key type .llcrypt unregistered [ 4605.926851] Key type ._llcrypt unregistered [ 4614.145820] Key type ._llcrypt registered [ 4614.147710] Key type .llcrypt registered [ 4614.196329] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 4621.311224] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 4621.703907] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4621.723928] alg: No test for adler32 (adler32-zlib) [ 4622.623437] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 4622.754388] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 4624.368294] Key type lgssc registered [ 4624.892452] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4639.858907] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4639.870795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4640.991840] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4641.016727] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4641.068663] Lustre: lustre-MDT0000: new disk, initializing [ 4641.103438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4641.111419] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4642.838126] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 4647.733731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4647.790618] Lustre: 149797:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4647.823412] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4647.826161] Lustre: Skipped 1 previous similar message [ 4647.872684] Lustre: lustre-MDT0001: new disk, initializing [ 4647.923676] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4647.940060] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4647.944499] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4649.759508] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 4654.221170] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4656.534901] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4659.677938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4659.831059] Lustre: lustre-OST0000: new disk, initializing [ 4659.834306] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4659.839682] Lustre: 151768:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4659.894524] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4662.739659] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 4666.921050] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4667.394118] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4667.398450] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4667.440913] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4670.501932] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_module obdecho/obdecho [ 4671.282336] LustreError: 152770:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 4671.286469] LustreError: 152770:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 4672.097289] LustreError: 152867:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 4677.604154] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4677.605660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4677.610014] Lustre: Skipped 1 previous similar message [ 4682.223239] Lustre: server umount lustre-MDT0000 complete [ 4683.640333] LustreError: 151767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753241 with bad export cookie 16334220249828488608 [ 4683.643894] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4683.646313] LustreError: 151767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4683.790433] Lustre: server umount lustre-MDT0001 complete [ 4695.668533] Lustre: server umount lustre-OST0000 complete [ 4705.104792] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 4709.499396] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 4726.923379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4727.079267] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4727.102380] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4727.150603] Lustre: lustre-MDT0000: new disk, initializing [ 4727.196142] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4727.213207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4729.414036] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 4735.464689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4735.548310] Lustre: 156573:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4735.583201] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4735.584743] Lustre: Skipped 1 previous similar message [ 4735.628392] Lustre: lustre-MDT0001: new disk, initializing [ 4735.686205] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4735.707673] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4735.712799] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4737.928451] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 4742.945923] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4745.969804] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4749.132729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4749.291509] Lustre: lustre-OST0000: new disk, initializing [ 4749.293985] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4749.297955] Lustre: 158546:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4749.354691] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4752.275335] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 4757.008074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4757.019717] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4757.109403] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4757.268858] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4759.518433] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4759.638640] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4761.770619] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4761.874656] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4767.208719] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4767.210219] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4767.211315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4767.211324] Lustre: Skipped 2 previous similar messages [ 4767.225059] Lustre: Skipped 2 previous similar messages [ 4768.997222] Lustre: server umount lustre-OST0000 complete [ 4771.810122] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4771.817408] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4771.829373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4771.833221] Lustre: Skipped 1 previous similar message [ 4777.361770] Lustre: server umount lustre-MDT0000 complete [ 4777.442560] LustreError: 157374:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4777.465525] LustreError: 157374:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 5 previous similar messages [ 4779.541562] LustreError: 156566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753337 with bad export cookie 16334220249828489553 [ 4779.546192] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4779.551429] LustreError: 156566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4779.765889] Lustre: server umount lustre-MDT0001 complete [ 4781.844578] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 23:55:38 (1788753338) [ 4782.725400] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 4783.784770] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 23:55:40 (1788753340) [ 4787.895400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4788.149713] LustreError: 160955:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4788.193847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4790.422809] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 4793.316813] LustreError: 160956:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4794.804250] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4796.902681] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 4800.429612] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4803.070301] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4806.629370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4809.747241] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 4813.478123] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4829.262454] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.8@tcp (stopping) [ 4829.265862] Lustre: Skipped 2 previous similar messages [ 4830.311311] Lustre: server umount lustre-MDT0000 complete [ 4830.691985] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4830.696295] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4830.697512] Lustre: Skipped 2 previous similar messages [ 4830.699876] LustreError: 160955:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4831.861643] LustreError: 160935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753389 with bad export cookie 16334220249828490414 [ 4831.866848] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4831.866973] LustreError: 160935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4834.382930] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.8@tcp (stopping) [ 4835.809193] LustreError: 163450:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4835.809520] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4835.816770] LustreError: 163450:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 4838.101121] Lustre: server umount lustre-MDT0001 complete [ 4840.942443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4846.547545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4846.650265] LustreError: 164812:0:(llog.c:1655:llog_backup()) MGC192.168.206.108@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 4846.654050] Lustre: 164812:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.108@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 4854.853429] LustreError: 164817:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4854.863627] LustreError: 164817:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 4871.649695] LustreError: 164817:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4871.657220] LustreError: 164817:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 9 previous similar messages [ 4905.028684] LustreError: 164838:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4905.036717] LustreError: 164838:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 23 previous similar messages [ 4908.000313] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 4908.005541] Lustre: 164812:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.108@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 4948.457428] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4948.464406] Lustre: Skipped 2 previous similar messages [ 4950.486653] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 4951.525668] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4954.482771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4969.542550] LustreError: 164819:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4969.550861] LustreError: 164819:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 45 previous similar messages [ 4975.012576] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 4977.056956] LustreError: 148023:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000d1209ee9 x1875643654581504/t0(0) o250->MGC192.168.206.108@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4979.385803] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 4982.249077] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4983.237793] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4984.056231] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4987.098104] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4987.849159] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4997.601369] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4997.603230] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4997.608041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4997.608054] Lustre: Skipped 2 previous similar messages [ 4997.630542] Lustre: Skipped 2 previous similar messages [ 4999.276851] Lustre: server umount lustre-MDT0000 complete [ 5000.882968] LustreError: 160937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753558 with bad export cookie 16334220249828496910 [ 5000.886077] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5000.891049] LustreError: 160937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5001.025217] Lustre: server umount lustre-MDT0001 complete [ 5013.034096] Lustre: server umount lustre-OST0000 complete [ 5021.504043] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 5025.832276] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 5043.723852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5043.853041] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5043.871249] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5043.915159] Lustre: lustre-MDT0000: new disk, initializing [ 5043.958155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5043.962721] Lustre: Skipped 1 previous similar message [ 5043.972741] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5046.104663] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5051.281846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5051.341509] Lustre: 169891:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5051.485237] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5053.227196] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5057.801996] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5060.410411] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5063.396428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5063.537879] Lustre: 171859:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5066.571352] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5066.633985] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5066.689374] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5071.502609] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5073.559599] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5073.675259] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5075.753141] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5075.867754] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5076.961417] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5076.963421] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5076.965844] LustreError: Skipped 1 previous similar message [ 5076.968098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5076.976803] Lustre: Skipped 1 previous similar message [ 5076.980582] Lustre: Skipped 2 previous similar messages [ 5091.297402] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5091.375208] Lustre: server umount lustre-OST0000 complete [ 5097.442793] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5099.320893] Lustre: server umount lustre-MDT0000 complete [ 5101.075053] LustreError: 169882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753658 with bad export cookie 16334220249828513710 [ 5101.077597] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5101.087934] LustreError: 169882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5101.278092] Lustre: server umount lustre-MDT0001 complete [ 5106.114805] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 00:01:02 (1788753662) [ 5109.746146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5109.964493] LustreError: 174283:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5109.974673] LustreError: 174283:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 10 previous similar messages [ 5112.088246] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5116.688252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5118.611457] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5121.904807] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5124.356894] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5127.606652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5130.595700] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5133.956960] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5148.206786] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 00:01:45 (1788753705) [ 5150.689673] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5150.693822] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5150.699188] Lustre: Skipped 2 previous similar messages [ 5150.701554] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5150.703908] Lustre: Skipped 7 previous similar messages [ 5155.549530] Lustre: server umount lustre-OST0000 complete [ 5163.441542] Lustre: server umount lustre-MDT0000 complete [ 5164.897941] LustreError: 174309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753722 with bad export cookie 16334220249828514592 [ 5164.898491] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5164.904717] LustreError: 174309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5165.044814] Lustre: server umount lustre-MDT0001 complete [ 5169.486409] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 5170.783964] Key type lgssc unregistered [ 5170.979897] LNet: 178028:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5170.984817] LNetError: 178028:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5171.001902] LNet: Removed LNI 192.168.206.108@tcp [ 5171.398106] Key type .llcrypt unregistered [ 5171.400199] Key type ._llcrypt unregistered [ 5179.922491] Key type ._llcrypt registered [ 5179.924325] Key type .llcrypt registered [ 5179.966936] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 5180.301663] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5180.310512] alg: No test for adler32 (adler32-zlib) [ 5181.173390] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 5181.273223] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 5182.856229] Key type lgssc registered [ 5183.270194] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5186.508155] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5186.517706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5201.952501] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5207.072381] LustreError: 179243:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.108@tcp: failed processing log, type 4: rc = -110 [ 5212.192469] LustreError: 179243:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 5212.198457] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 5212.200905] LustreError: MGC192.168.206.108@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 5212.205880] LustreError: 179243:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-OST0000: -22 [ 5212.210300] LustreError: 179243:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -22 [ 5212.213558] LustreError: 179243:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 5212.241467] Lustre: server umount lustre-OST0000 complete [ 5212.243633] LustreError: 179243:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5215.666936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5216.859843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5218.504090] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5222.626648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5222.817362] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5224.732618] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5228.259859] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5230.791555] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5233.915131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5234.037381] LustreError: 181557:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 5234.044800] Lustre: 181557:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.206.108@tcp: invalid local config log lustre-OST0000: rc = -22 [ 5234.112942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5236.736555] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5239.787898] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5243.360884] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5243.364404] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5243.370633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5244.385981] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5244.387666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5244.391155] Lustre: Skipped 1 previous similar message [ 5244.393490] Lustre: Skipped 1 previous similar message [ 5246.813303] Lustre: server umount lustre-MDT0000 complete [ 5247.459221] LustreError: 180941:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5247.467158] LustreError: 180941:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 4 previous similar messages [ 5248.341404] LustreError: 180476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753806 with bad export cookie 6140068232434348846 [ 5248.344830] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5248.345788] LustreError: 180476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5249.504306] LustreError: 180470:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000064e914c8 x1875644240647424/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5249.570815] Lustre: server umount lustre-MDT0001 complete [ 5261.471449] Lustre: server umount lustre-OST0000 complete [ 5264.986360] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 5268.921490] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 5283.168795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5283.289496] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5283.303854] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5283.345521] Lustre: lustre-MDT0000: new disk, initializing [ 5283.384238] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5283.393795] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5285.115816] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5289.876057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5289.926764] Lustre: 185075:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5289.954297] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5289.957211] Lustre: Skipped 1 previous similar message [ 5289.998041] Lustre: lustre-MDT0001: new disk, initializing [ 5290.038777] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5290.059290] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5290.066088] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5291.708993] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5296.070661] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5298.333456] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5301.235289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5301.365715] Lustre: lustre-OST0000: new disk, initializing [ 5301.367936] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5301.371797] Lustre: 187043:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5301.420322] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5304.198134] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5305.020691] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5305.025258] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5305.058642] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5308.668125] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5310.408963] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5310.508429] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5312.321036] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5312.429371] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5315.560221] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5315.560816] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5315.561715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5315.565529] Lustre: Skipped 1 previous similar message [ 5315.569154] LustreError: Skipped 1 previous similar message [ 5319.383762] Lustre: server umount lustre-OST0000 complete [ 5325.793947] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5325.793990] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5325.796900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5325.800601] Lustre: Skipped 1 previous similar message [ 5325.803037] Lustre: Skipped 1 previous similar message [ 5326.949936] Lustre: server umount lustre-MDT0000 complete [ 5328.524462] LustreError: 187042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753886 with bad export cookie 6140068232434349560 [ 5328.528629] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5328.529966] LustreError: 187042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5328.671727] Lustre: server umount lustre-MDT0001 complete [ 5332.109257] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 00:04:49 (1788753889) [ 5338.917969] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 5340.109143] Key type lgssc unregistered [ 5340.244650] LNet: 189983:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5340.248194] LNetError: 189983:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5340.255496] LNet: Removed LNI 192.168.206.108@tcp [ 5340.637521] Key type .llcrypt unregistered [ 5340.639477] Key type ._llcrypt unregistered [ 5348.456241] Key type ._llcrypt registered [ 5348.457456] Key type .llcrypt registered [ 5348.503361] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 5348.914614] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5348.980964] alg: No test for adler32 (adler32-zlib) [ 5349.853660] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 5349.959553] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 5351.552217] Key type lgssc registered [ 5352.007142] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5355.120334] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5355.129095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5356.309272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5358.052365] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5361.491946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5361.668885] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5363.272084] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5365.964310] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5368.147787] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5371.338806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5371.434977] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5371.437136] LustreError: 193030:0:(llog.c:1696:llog_backup()) MGC192.168.206.108@tcp: failed to backup log lustre-OST0000: rc = -28 [ 5371.441694] Lustre: 193030:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.206.108@tcp: can't backup local config lustre-OST0000: rc = -28 [ 5371.493175] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5374.023677] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5376.986499] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5377.872060] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5377.874133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5377.876723] Lustre: Skipped 2 previous similar messages [ 5377.880844] Lustre: Skipped 1 previous similar message [ 5383.138951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5383.141574] Lustre: Skipped 3 previous similar messages [ 5388.258140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5388.261591] Lustre: Skipped 2 previous similar messages [ 5392.352127] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5392.418205] Lustre: server umount lustre-MDT0000 complete [ 5393.380224] LustreError: 191172:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5393.387469] LustreError: 191172:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 4 previous similar messages [ 5393.720356] LustreError: 191154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753951 with bad export cookie 9100162034950334657 [ 5393.722976] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5393.725079] LustreError: 191154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5393.841604] Lustre: server umount lustre-MDT0001 complete [ 5405.485318] Lustre: server umount lustre-OST0000 complete [ 5409.511919] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 5413.751628] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 5427.774236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5427.884961] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5427.900603] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5427.947133] Lustre: lustre-MDT0000: new disk, initializing [ 5427.984619] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5427.993657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5429.665878] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5435.139073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5435.193804] Lustre: 196550:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5435.217390] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5435.219187] Lustre: Skipped 1 previous similar message [ 5435.221058] LustreError: 197332:0:(llog.c:1696:llog_backup()) MGC192.168.206.108@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 5435.225568] LustreError: 197332:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 5435.227819] Lustre: 197332:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.108@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 5435.236685] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5435.239231] Lustre: Skipped 1 previous similar message [ 5435.281326] Lustre: lustre-MDT0001: new disk, initializing [ 5435.330600] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5435.348070] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5435.353189] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5437.160649] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5441.857821] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5444.060919] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5446.944534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5447.067810] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5447.069430] Lustre: Skipped 1 previous similar message [ 5447.071270] LustreError: 198496:0:(llog.c:1696:llog_backup()) MGC192.168.206.108@tcp: failed to backup log lustre-OST0000: rc = -28 [ 5447.077424] LustreError: 198496:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 5447.082217] Lustre: 198496:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.108@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 5447.088459] Lustre: 198496:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 5447.102771] Lustre: lustre-OST0000: new disk, initializing [ 5447.107399] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5447.111599] Lustre: 198521:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5447.160088] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5449.159543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5449.164261] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5449.207577] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5449.695892] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5454.204935] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5456.274249] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5456.392917] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5458.422191] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5458.532440] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5459.426019] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5459.426821] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5459.427882] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5459.431858] Lustre: Skipped 1 previous similar message [ 5464.548311] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5464.551366] Lustre: Skipped 2 previous similar messages [ 5465.569879] Lustre: server umount lustre-OST0000 complete [ 5469.666294] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5469.671628] Lustre: Skipped 1 previous similar message [ 5473.325263] Lustre: server umount lustre-MDT0000 complete [ 5474.786443] LustreError: 197363:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5474.795937] LustreError: 197363:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 5474.951338] LustreError: 196545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788754032 with bad export cookie 9100162034950335343 [ 5474.954622] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5474.956060] LustreError: 196545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5475.084398] Lustre: server umount lustre-MDT0001 complete [ 5478.531694] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 00:07:15 (1788754035) [ 5485.327232] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 5489.680377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5489.885180] LustreError: 201648:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5489.914570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5491.709615] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5495.266031] LustreError: 201649:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5495.571752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5497.535369] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5498.907576] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5501.753151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5504.355331] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5507.890304] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5507.988516] Lustre: lustre-OST0001: new disk, initializing [ 5507.991031] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5507.996634] Lustre: 203958:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5510.537464] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5513.198678] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5513.202854] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5513.223487] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5520.740161] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5529.072778] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5529.943479] Lustre: DEBUG MARKER: ADD OST3 [ 5532.212142] loop: module loaded [ 5532.238305] loop0: detected capacity change from 0 to 204800000 [ 5532.327014] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5532.334033] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5532.343018] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5535.048422] loop0: detected capacity change from 0 to 204800000 [ 5537.506287] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5537.601649] Lustre: lustre-OST0003: new disk, initializing [ 5537.604281] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 5537.607892] Lustre: 205712:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5537.625830] Lustre: DEBUG MARKER: STOP OST3 [ 5537.672541] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 5537.675405] Lustre: Skipped 3 previous similar messages [ 5538.476675] Lustre: Failing over lustre-OST0003 [ 5538.519293] Lustre: server umount lustre-OST0003 complete [ 5538.850737] LustreError: 203136:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.206.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5540.503342] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5546.067924] Lustre: DEBUG MARKER: START OST3 again [ 5548.958572] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5551.752594] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5554.166600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 5554.171663] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 5554.190217] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000401 [ 5566.439739] Lustre: Failing over lustre-OST0003 [ 5566.494754] Lustre: server umount lustre-OST0003 complete [ 5569.507466] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5569.511065] LustreError: Skipped 1 previous similar message [ 5569.513264] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5569.516623] LustreError: 204793:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5569.519486] Lustre: Skipped 1 previous similar message [ 5569.526632] LustreError: 204793:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 6 previous similar messages [ 5579.750901] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5579.752762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5579.757193] Lustre: Skipped 3 previous similar messages [ 5579.759738] Lustre: Skipped 4 previous similar messages [ 5585.712598] Lustre: server umount lustre-MDT0000 complete [ 5587.349179] LustreError: 202782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788754145 with bad export cookie 9100162034950336225 [ 5587.353651] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5587.353820] LustreError: 202782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5589.986975] LustreError: 202380:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5589.988055] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5590.000490] LustreError: 202380:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 5 previous similar messages [ 5590.006148] Lustre: Skipped 1 previous similar message [ 5593.606534] Lustre: server umount lustre-MDT0001 complete [ 5595.172654] Lustre: server umount lustre-OST0000 complete [ 5596.762444] Lustre: server umount lustre-OST0001 complete [ 5599.415582] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 5602.604523] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 5615.986170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5616.076069] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5616.090525] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5616.127717] Lustre: lustre-MDT0000: new disk, initializing [ 5616.149105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5616.151301] Lustre: Skipped 1 previous similar message [ 5616.156885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5617.667435] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5622.144711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5622.192376] Lustre: 211243:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5622.306430] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5623.794047] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5627.999852] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5630.142015] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5632.755080] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5632.891253] Lustre: lustre-OST0000: new disk, initializing [ 5632.893343] Lustre: Skipped 1 previous similar message [ 5632.895578] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5632.898474] Lustre: Skipped 2 previous similar messages [ 5632.901160] Lustre: 213211:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5634.626234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5634.629651] Lustre: Skipped 1 previous similar message [ 5634.632215] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5634.671392] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5635.141298] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5639.176687] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5640.739130] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5640.814732] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5642.365745] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5642.446274] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5644.769796] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5644.770375] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5644.773880] LustreError: Skipped 2 previous similar messages [ 5644.779737] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5644.783301] Lustre: Skipped 10 previous similar messages [ 5649.316817] Lustre: server umount lustre-OST0000 complete [ 5652.961724] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5665.248115] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5665.315249] Lustre: server umount lustre-MDT0000 complete [ 5666.663276] LustreError: 211234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788754224 with bad export cookie 9100162034950338486 [ 5666.668435] LustreError: 211234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5666.668800] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5666.794144] Lustre: server umount lustre-MDT0001 complete [ 5668.169064] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 00:10:25 (1788754225) [ 5674.160846] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 5678.079874] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 5692.088662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5692.201219] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5692.218373] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5692.257632] Lustre: lustre-MDT0000: new disk, initializing [ 5692.304896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5694.071186] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5698.843362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5698.896476] Lustre: 217814:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5699.021052] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5700.654055] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5704.900690] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5707.208374] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5710.147044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5710.292148] Lustre: 219784:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5711.410189] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5712.712631] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5717.377816] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5719.200728] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5719.296159] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5721.032457] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5721.129390] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5726.690475] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5726.692115] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5726.692825] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5726.692829] Lustre: Skipped 7 previous similar messages [ 5726.697023] Lustre: Skipped 4 previous similar messages [ 5726.700555] LustreError: Skipped 1 previous similar message [ 5728.042230] Lustre: server umount lustre-OST0000 complete [ 5735.712287] Lustre: server umount lustre-MDT0000 complete [ 5736.931833] LustreError: 217825:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5736.937933] LustreError: 217825:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 5737.233693] LustreError: 219783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788754295 with bad export cookie 9100162034950339368 [ 5737.236338] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5737.240067] LustreError: 219783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5737.384810] Lustre: server umount lustre-MDT0001 complete [ 5740.872768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5742.907307] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5746.624081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5746.811476] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5746.815204] Lustre: Skipped 6 previous similar messages [ 5748.560511] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5751.465587] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5753.593239] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5756.311500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5758.601221] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5761.401177] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5775.842211] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5775.845738] LustreError: Skipped 1 previous similar message [ 5779.669096] Lustre: server umount lustre-OST0000 complete [ 5787.296046] Lustre: server umount lustre-MDT0000 complete [ 5788.742310] LustreError: 221975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788754346 with bad export cookie 9100162034950340250 [ 5788.746071] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5788.748523] LustreError: 221975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5788.885742] Lustre: server umount lustre-MDT0001 complete [ 5896.905263] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 5898.155293] Key type lgssc unregistered [ 5898.308750] LNet: 225674:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5898.312432] LNetError: 225674:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5898.327336] LNet: Removed LNI 192.168.206.108@tcp [ 5898.722732] Key type .llcrypt unregistered [ 5898.726036] Key type ._llcrypt unregistered [ 5908.896357] Key type ._llcrypt registered [ 5908.897906] Key type .llcrypt registered [ 5908.947302] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 5916.252175] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 5916.728903] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5916.741422] alg: No test for adler32 (adler32-zlib) [ 5917.615233] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 5917.716970] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 5919.305336] Key type lgssc registered [ 5919.741311] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5934.446400] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5934.456214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5935.562564] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5935.584319] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5935.628065] Lustre: lustre-MDT0000: new disk, initializing [ 5935.662291] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5935.673845] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5937.423073] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5942.536232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5942.595615] Lustre: 229010:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5942.634322] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5942.638250] Lustre: Skipped 1 previous similar message [ 5942.679721] Lustre: lustre-MDT0001: new disk, initializing [ 5942.728790] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5942.744227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5942.749150] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5944.432960] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5948.892412] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5951.061890] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5953.774298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5953.888851] Lustre: lustre-OST0000: new disk, initializing [ 5953.891209] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5953.895276] Lustre: 230980:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5953.929198] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5956.288128] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5956.432685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5956.436730] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5956.479541] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5960.551986] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5962.280404] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5962.384611] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5964.039969] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5964.127132] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5966.821041] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5966.822228] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5966.829795] LustreError: Skipped 1 previous similar message [ 5966.831552] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5966.839648] Lustre: Skipped 1 previous similar message [ 5971.937971] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5971.940894] Lustre: Skipped 2 previous similar messages [ 5977.059259] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5977.062771] Lustre: Skipped 1 previous similar message [ 5979.104136] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5979.179826] Lustre: server umount lustre-OST0000 complete [ 5982.178444] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5982.179807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5982.183872] Lustre: Skipped 1 previous similar message [ 5986.852230] Lustre: server umount lustre-MDT0000 complete [ 5988.406395] LustreError: 229803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788754546 with bad export cookie 16587984984788188713 [ 5988.410326] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5988.415198] LustreError: 229803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5988.551312] Lustre: server umount lustre-MDT0001 complete [ 5995.657512] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 5999.767966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5999.936731] LustreError: 233910:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5999.946410] LustreError: 233910:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 5999.959907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6001.639588] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6005.141098] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6005.217100] LustreError: 233911:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6005.312064] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6006.886810] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6008.028127] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6010.869969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6013.387791] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6016.937342] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6017.013596] Lustre: lustre-OST0001: new disk, initializing [ 6017.015699] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6017.019314] Lustre: 236215:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6017.064432] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6017.067555] Lustre: Skipped 1 previous similar message [ 6019.447643] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6024.202650] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6024.206989] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6024.238050] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6029.599786] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6032.563346] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6037.750932] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 00:16:34 (1788754594) [ 6040.548958] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6040.553159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6040.556591] Lustre: Skipped 2 previous similar messages [ 6040.567971] Lustre: Skipped 4 previous similar messages [ 6040.866313] Lustre: server umount lustre-MDT0000 complete [ 6042.371772] LustreError: 236214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788754600 with bad export cookie 16587984984788189595 [ 6042.376614] LustreError: 236214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6042.377590] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6042.504384] Lustre: server umount lustre-MDT0001 complete [ 6053.892285] Lustre: server umount lustre-OST0000 complete [ 6065.567120] Lustre: server umount lustre-OST0001 complete [ 6067.899364] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 6071.695750] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 6085.580714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6085.690193] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6085.704288] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6085.745687] Lustre: lustre-MDT0000: new disk, initializing [ 6085.785973] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6085.794190] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6087.463743] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6092.165217] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6092.211253] Lustre: 240158:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6092.337110] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6094.000574] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6098.288043] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6100.405960] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6103.023817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6103.152356] Lustre: lustre-OST0000: new disk, initializing [ 6103.154689] Lustre: Skipped 1 previous similar message [ 6103.156910] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6103.159011] Lustre: Skipped 2 previous similar messages [ 6103.161681] Lustre: 242126:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6104.258902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6104.263056] Lustre: Skipped 1 previous similar message [ 6104.265623] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6104.305065] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6105.473669] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6109.748845] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6111.460307] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6111.554749] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6113.262727] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6113.358956] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6114.787678] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6114.789164] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6114.796924] Lustre: Skipped 1 previous similar message [ 6114.804573] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6114.807644] Lustre: Skipped 1 previous similar message [ 6120.327530] Lustre: server umount lustre-OST0000 complete [ 6122.977736] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6122.982673] LustreError: Skipped 1 previous similar message [ 6122.985770] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6122.992677] Lustre: Skipped 1 previous similar message [ 6128.363862] Lustre: server umount lustre-MDT0000 complete [ 6129.978762] LustreError: 240953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788754687 with bad export cookie 16587984984788191716 [ 6129.980545] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6129.984962] LustreError: 240953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6130.132081] Lustre: server umount lustre-MDT0001 complete [ 6133.551772] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6133.749587] LustreError: 244340:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6133.783736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6133.787025] Lustre: Skipped 2 previous similar messages [ 6135.297386] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6138.617374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6140.555529] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6143.929613] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6146.547788] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6149.697561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6152.584239] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6156.304204] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6301.666121] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6301.667106] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6301.677619] Lustre: Skipped 1 previous similar message [ 6301.689738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6301.692448] Lustre: Skipped 5 previous similar messages [ 6305.693334] Lustre: server umount lustre-OST0000 complete [ 6307.809163] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6313.451354] Lustre: server umount lustre-MDT0000 complete [ 6315.008287] LustreError: 244321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788754872 with bad export cookie 16587984984788192598 [ 6315.012188] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6315.018425] LustreError: 244321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6315.159611] Lustre: server umount lustre-MDT0001 complete [ 6429.797367] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 6431.167920] Key type lgssc unregistered [ 6431.333046] LNet: 247945:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6431.338402] LNetError: 247945:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6431.353713] LNet: Removed LNI 192.168.206.108@tcp [ 6431.796106] Key type .llcrypt unregistered [ 6431.797894] Key type ._llcrypt unregistered [ 6443.272407] Key type ._llcrypt registered [ 6443.273833] Key type .llcrypt registered [ 6443.319686] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 6450.792981] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 6451.220561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6451.239965] alg: No test for adler32 (adler32-zlib) [ 6452.125578] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 6452.256460] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 6453.872153] Key type lgssc registered [ 6454.387391] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6473.472501] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 6478.303576] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6478.324551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6479.448425] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6479.465982] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6479.514380] Lustre: lustre-MDT0000: new disk, initializing [ 6479.593511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6479.603627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6481.805325] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6487.462694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6487.520922] Lustre: 251986:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6487.561515] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6487.564480] Lustre: Skipped 1 previous similar message [ 6487.612564] Lustre: lustre-MDT0001: new disk, initializing [ 6487.658813] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6487.674657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6487.683144] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6489.461305] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6492.142441] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6494.980296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6495.120330] Lustre: lustre-OST0000: new disk, initializing [ 6495.122892] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6495.126458] Lustre: 253617:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6495.180272] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6497.702214] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6501.390937] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6501.398711] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6501.447395] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6502.876328] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6502.979363] Lustre: lustre-OST0001: new disk, initializing [ 6502.982441] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6502.986446] Lustre: 254486:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6503.044518] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6505.532308] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6509.069429] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6509.074825] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6509.107891] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6511.508722] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6515.383975] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6519.267824] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6519.271045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6519.273557] Lustre: Skipped 3 previous similar messages [ 6519.276172] Lustre: Skipped 2 previous similar messages [ 6524.386800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6524.389958] Lustre: Skipped 2 previous similar messages [ 6525.311619] Lustre: server umount lustre-MDT0000 complete [ 6526.812183] LustreError: 251976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788755084 with bad export cookie 17868696500103211278 [ 6526.815549] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6526.817599] LustreError: 251976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6526.949725] Lustre: server umount lustre-MDT0001 complete [ 6538.766561] Lustre: server umount lustre-OST0000 complete [ 6549.290650] Lustre: server umount lustre-OST0001 complete [ 6554.306312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6557.706061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6560.800771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6564.639433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6571.888944] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 6576.320831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6576.331518] Lustre: lustre-MDT0000: reset Object Index mappings [ 6576.563329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6578.335557] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6581.728978] LustreError: 259489:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6581.736424] LustreError: 259489:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 6582.141764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6582.150458] Lustre: lustre-MDT0001: reset Object Index mappings [ 6584.008589] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6585.264362] Lustre: 260631:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6588.151746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6588.359673] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6588.367017] Lustre: Skipped 1 previous similar message [ 6590.761697] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6592.486318] LustreError: 260983:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6594.316045] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6596.662575] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6601.706178] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6601.708880] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 6605.719211] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6607.215871] Lustre: 262498:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6611.938370] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6611.941762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6611.943874] Lustre: Skipped 2 previous similar messages [ 6611.949336] Lustre: Skipped 3 previous similar messages [ 6614.721243] Lustre: server umount lustre-MDT0000 complete [ 6616.498595] LustreError: 260234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788755174 with bad export cookie 17868696500103218068 [ 6616.500488] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6616.504051] LustreError: 260234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6616.651443] Lustre: server umount lustre-MDT0001 complete [ 6628.890431] Lustre: server umount lustre-OST0000 complete [ 6641.079176] Lustre: server umount lustre-OST0001 complete [ 6649.676295] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 00:26:46 (1788755206) [ 6657.010957] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 6661.586977] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 6676.308922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6676.409520] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6676.422309] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6676.463694] Lustre: lustre-MDT0000: new disk, initializing [ 6676.503647] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6676.507115] Lustre: Skipped 1 previous similar message [ 6676.515493] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6678.260836] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6683.210454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6683.263750] Lustre: 266748:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6683.392796] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6685.196900] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6689.496336] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6691.593992] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6694.291062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6694.400431] Lustre: lustre-OST0000: new disk, initializing [ 6694.402095] Lustre: Skipped 1 previous similar message [ 6694.404065] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6694.407149] Lustre: Skipped 2 previous similar messages [ 6694.409491] Lustre: 268714:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6695.501479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6695.505374] Lustre: Skipped 1 previous similar message [ 6695.507822] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6695.552202] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6696.838612] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6701.115139] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6702.754214] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6702.844950] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6704.476650] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6704.563693] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6706.148639] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6706.148857] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6706.149574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6706.153966] Lustre: Skipped 2 previous similar messages [ 6706.162209] LustreError: Skipped 1 previous similar message [ 6711.266707] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6711.268753] Lustre: Skipped 2 previous similar messages [ 6711.384338] Lustre: server umount lustre-OST0000 complete [ 6714.336865] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6714.339944] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6721.505039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6721.507851] Lustre: Skipped 4 previous similar messages [ 6727.136126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6727.199112] Lustre: server umount lustre-MDT0000 complete [ 6728.614062] LustreError: 267543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788755286 with bad export cookie 17868696500103220154 [ 6728.616699] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6728.619822] LustreError: 267543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6728.771082] Lustre: server umount lustre-MDT0001 complete [ 6736.218519] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 6739.923346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6740.059456] LustreError: 271642:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6740.072258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6740.074956] Lustre: Skipped 2 previous similar messages [ 6741.619393] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6744.810880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6746.384229] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6747.447827] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6749.732207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6752.088534] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6754.985036] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6755.058940] Lustre: lustre-OST0001: new disk, initializing [ 6755.061288] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6755.064900] Lustre: 273950:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6756.687859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6756.692049] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6756.735873] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6757.490546] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6762.293582] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6763.542980] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6765.427739] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 6765.453417] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 6770.657397] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6770.661083] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6770.664937] Lustre: Skipped 1 previous similar message [ 6770.667274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6770.670050] Lustre: Skipped 2 previous similar messages [ 6774.691339] Lustre: server umount lustre-MDT0000 complete [ 6775.978538] LustreError: 274715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788755333 with bad export cookie 17868696500103221036 [ 6775.982789] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6775.983273] LustreError: 274715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6776.097407] Lustre: server umount lustre-MDT0001 complete [ 6787.653076] Lustre: server umount lustre-OST0000 complete [ 6799.194535] Lustre: server umount lustre-OST0001 complete [ 6805.421675] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 6809.976658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6810.154225] LustreError: 277291:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6810.182535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6810.186399] Lustre: Skipped 3 previous similar messages [ 6811.792512] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6815.203695] LustreError: 277292:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6815.370803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6817.152699] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6818.320509] Lustre: 278433:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6820.970829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6823.568485] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6825.258074] LustreError: 278786:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6827.207459] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6829.617730] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6835.698118] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 6835.707100] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 6838.651682] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6840.169345] Lustre: 280312:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6842.271101] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 6842.294891] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 6846.964528] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 00:30:03 (1788755403) [ 6861.283585] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6861.287336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6861.289665] Lustre: Skipped 6 previous similar messages [ 6861.293711] Lustre: Skipped 7 previous similar messages [ 6864.800724] Lustre: server umount lustre-MDT0000 complete [ 6866.275747] LustreError: 280314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788755424 with bad export cookie 17868696500103227392 [ 6866.280194] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6866.282966] LustreError: 280314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6866.401666] LustreError: 278749:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6866.411476] LustreError: 278749:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 6872.463699] Lustre: server umount lustre-MDT0001 complete [ 6880.231475] Lustre: server umount lustre-OST0000 complete [ 6888.179350] Lustre: server umount lustre-OST0001 complete [ 6891.134375] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 6895.036526] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 6912.812280] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 6917.444050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6917.556230] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6917.572786] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6917.612414] Lustre: lustre-MDT0000: new disk, initializing [ 6917.662960] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6919.518622] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6924.495430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6924.554213] Lustre: 284827:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6924.688671] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6926.702406] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6932.338076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6932.472415] Lustre: 286462:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6934.331483] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6934.866392] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6940.026538] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6940.152361] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6940.155165] Lustre: Skipped 6 previous similar messages [ 6941.246787] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6942.747327] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6947.975766] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6954.925236] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 00:31:51 (1788755511) [ 6956.518599] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6956.520323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6956.524046] Lustre: Skipped 5 previous similar messages [ 6956.526639] Lustre: Skipped 6 previous similar messages [ 6961.892268] Lustre: server umount lustre-MDT0000 complete [ 6963.214588] LustreError: 284821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788755521 with bad export cookie 17868696500103235351 [ 6963.217421] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6963.220413] LustreError: 284821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6963.345773] Lustre: server umount lustre-MDT0001 complete [ 6974.988776] Lustre: server umount lustre-OST0000 complete [ 6986.685691] Lustre: server umount lustre-OST0001 complete [ 6989.869937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6990.078120] LustreError: 289573:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6990.086695] LustreError: 289573:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 6991.763464] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6994.953257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6996.719825] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6999.636384] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7001.746555] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7004.361228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7006.854212] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7009.394765] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7025.749305] loop1: detected capacity change from 0 to 204800000 [ 7025.845815] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7025.853030] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7025.859435] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7027.924826] loop1: detected capacity change from 0 to 204800000 [ 7030.162217] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7030.201565] LustreError: 289563:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs6960: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 7030.208162] LustreError: 289563:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs6960 [ 7030.211452] LustreError: 289563:0:(mgs_handler.c:517:mgs_target_reg()) fs6960-MDT0001: Error checking target index: rc = -13 [ 7030.217516] LustreError: fs6960-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 7030.222147] LustreError: 293309:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [ 7030.225412] LustreError: 293309:0:(tgt_mount.c:2128:server_put_super()) no obd fs6960-MDT0001 [ 7030.227934] LustreError: 293309:0:(tgt_mount.c:116:server_deregister_mount()) fs6960-MDT0001 not registered [ 7030.250486] Lustre: server umount fs6960-MDT0001 complete [ 7030.252761] LustreError: 293309:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 7033.499511] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7033.534930] Lustre: Setting parameter fs6960-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs6960-MDT0001 [ 7033.541945] Lustre: Skipped 2 previous similar messages [ 7033.560906] Lustre: srv-fs6960-MDT0001: No data found on store. Initialize space. [ 7033.563933] Lustre: Skipped 4 previous similar messages [ 7033.605300] Lustre: fs6960-MDT0001: new disk, initializing [ 7033.607233] Lustre: Skipped 3 previous similar messages [ 7033.647042] Lustre: srv-fs6960-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 7033.650432] Lustre: 293776:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs6960-MDT0001-mdtlov: can't create new llog: rc = -115 [ 7033.653774] LustreError: 293776:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs6960-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 7035.176061] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7038.277104] Lustre: server umount fs6960-MDT0001 complete [ 7043.373663] loop1: detected capacity change from 0 to 204800000 [ 7045.539023] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7045.661953] Lustre: 295343:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs6960-MDT0001-mdtlov: can't create new llog: rc = -115 [ 7045.665675] LustreError: 295343:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs6960-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 7047.241471] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7049.137550] Lustre: server umount fs6960-MDT0001 complete [ 7055.843077] LustreError: 292072:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7055.843383] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7055.849646] LustreError: 292072:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 23 previous similar messages [ 7055.855306] Lustre: Skipped 2 previous similar messages [ 7067.104241] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7067.171941] Lustre: server umount lustre-MDT0000 complete [ 7070.268252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7070.328898] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7072.055514] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7075.372865] loop1: detected capacity change from 0 to 204800000 [ 7075.817946] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7077.747455] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7077.890826] Lustre: 297784:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs6960-MDT0001-mdtlov: can't create new llog: rc = -115 [ 7077.895034] LustreError: 297784:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs6960-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 7079.520571] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7081.533612] Lustre: server umount fs6960-MDT0001 complete [ 7091.171535] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7091.176261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7091.178874] Lustre: Skipped 17 previous similar messages [ 7095.572813] Lustre: server umount lustre-OST0000 complete [ 7101.410023] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7103.135220] Lustre: server umount lustre-MDT0000 complete [ 7104.530171] LustreError: 290316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788755662 with bad export cookie 17868696500103238025 [ 7104.533161] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7104.534413] LustreError: 290316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7104.660075] Lustre: server umount lustre-MDT0001 complete [ 7108.619265] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 7109.782553] Key type lgssc unregistered [ 7109.917482] LNet: 300125:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7109.920820] LNetError: 300125:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7109.929283] LNet: Removed LNI 192.168.206.108@tcp [ 7110.270148] Key type .llcrypt unregistered [ 7110.271373] Key type ._llcrypt unregistered [ 7118.570123] Key type ._llcrypt registered [ 7118.571330] Key type .llcrypt registered [ 7118.610737] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 7118.953127] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7119.014586] alg: No test for adler32 (adler32-zlib) [ 7119.876962] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 7119.974835] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 7121.560150] Key type lgssc registered [ 7121.941430] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7124.662817] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7124.669719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7125.842791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7127.279311] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7130.427343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7130.606189] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7132.183147] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7135.100987] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7137.179570] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7139.717299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7139.853931] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7140.900555] LustreError: 303204:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7140.908651] LustreError: 303204:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 7141.924165] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7144.538287] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7145.957843] LustreError: 303204:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7151.079772] LustreError: 303201:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7151.092014] LustreError: 303201:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 7155.285183] LustreError: 303385:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7160.392615] LustreError: 303203:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7160.405394] LustreError: 303203:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 7162.851900] loop1: detected capacity change from 0 to 204800000 [ 7162.919243] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7162.924073] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7162.929392] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7164.850116] loop1: detected capacity change from 0 to 204800000 [ 7166.982237] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7167.042670] Lustre: fs25619-OST0000: new disk, initializing [ 7167.045579] Lustre: srv-fs25619-OST0000: No data found on store. Initialize space. [ 7167.049615] Lustre: 305127:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7167.078488] Lustre: fs25619-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7169.337345] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7170.631194] LustreError: 303201:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7170.641366] LustreError: 303201:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 5 previous similar messages [ 7173.145163] Lustre: server umount fs25619-OST0000 complete [ 7177.551310] loop1: detected capacity change from 0 to 204800000 [ 7179.566652] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7179.638175] Lustre: fs25619-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7181.730853] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7184.987771] loop2: detected capacity change from 0 to 204800000 [ 7185.067380] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7185.072053] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7185.076893] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7186.920369] LustreError: 303385:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7186.928365] LustreError: 303385:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 10 previous similar messages [ 7187.149453] loop2: detected capacity change from 0 to 204800000 [ 7189.404131] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7189.445135] LustreError: fs25619-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 7189.450127] LustreError: fs25619-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 7189.454587] LustreError: 308163:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [ 7189.458078] LustreError: 308163:0:(tgt_mount.c:2128:server_put_super()) no obd fs25619-OST0001 [ 7189.460952] LustreError: 308163:0:(tgt_mount.c:116:server_deregister_mount()) fs25619-OST0001 not registered [ 7189.475534] Lustre: server umount fs25619-OST0001 complete [ 7189.477531] LustreError: 308163:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 7193.213551] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7193.256695] Lustre: fs25619-OST0001: new disk, initializing [ 7193.258798] Lustre: srv-fs25619-OST0001: No data found on store. Initialize space. [ 7193.261663] Lustre: 308714:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7193.286194] Lustre: fs25619-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7195.361936] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7198.900087] Lustre: server umount fs25619-OST0000 complete [ 7202.182430] Lustre: server umount fs25619-OST0001 complete [ 7206.339671] loop1: detected capacity change from 0 to 204800000 [ 7208.421347] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7210.767184] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7213.367170] loop2: detected capacity change from 0 to 204800000 [ 7215.392333] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7215.463330] Lustre: fs25619-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7215.466106] Lustre: Skipped 1 previous similar message [ 7217.529286] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7220.811618] Lustre: server umount fs25619-OST0000 complete [ 7221.829625] LustreError: 303385:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7221.836811] LustreError: 303385:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 18 previous similar messages [ 7224.358304] Lustre: server umount fs25619-OST0001 complete [ 7232.998668] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7232.999220] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7233.001332] LustreError: Skipped 1 previous similar message [ 7233.006990] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7233.010111] Lustre: Skipped 1 previous similar message [ 7234.718911] Lustre: server umount lustre-OST0000 complete [ 7238.114022] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7238.114233] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7238.114899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7238.117015] LustreError: Skipped 1 previous similar message [ 7238.127820] Lustre: Skipped 2 previous similar messages [ 7242.272580] Lustre: server umount lustre-MDT0000 complete [ 7243.670518] LustreError: 301356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788755801 with bad export cookie 1647542571761302230 [ 7243.673666] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7243.675816] LustreError: 301356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7243.813662] Lustre: server umount lustre-MDT0001 complete [ 7247.696783] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 7248.853889] Key type lgssc unregistered [ 7248.991666] LNet: 314624:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7248.995801] LNetError: 314624:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7249.006383] LNet: Removed LNI 192.168.206.108@tcp [ 7249.386120] Key type .llcrypt unregistered [ 7249.387391] Key type ._llcrypt unregistered [ 7260.945420] Key type ._llcrypt registered [ 7260.947011] Key type .llcrypt registered [ 7260.987508] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 7267.851138] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 7268.279840] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7268.321507] alg: No test for adler32 (adler32-zlib) [ 7269.199675] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 7269.304493] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 7270.896146] Key type lgssc registered [ 7271.311607] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7285.318113] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7285.327265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7286.420967] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7286.435290] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7286.472306] Lustre: lustre-MDT0000: new disk, initializing [ 7286.499779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7286.509860] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7288.094798] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7292.866165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7292.928143] Lustre: 318453:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7292.970945] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7292.974849] Lustre: Skipped 1 previous similar message [ 7293.016306] Lustre: lustre-MDT0001: new disk, initializing [ 7293.055922] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7293.072332] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7293.076856] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7294.682589] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7298.757872] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7300.860148] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7303.658302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7303.785304] Lustre: lustre-OST0000: new disk, initializing [ 7303.787738] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7303.790833] Lustre: 320418:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7303.832368] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7304.972257] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7304.977514] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7305.044704] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7306.259958] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7310.509379] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7311.360573] Lustre: Mounted lustre-client - version 2.17.58_41_g2290de3 [ 7311.740100] Lustre: Unmounted lustre-client [ 7315.425604] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7315.430744] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7315.437580] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7315.440246] Lustre: Skipped 1 previous similar message [ 7316.452703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7316.458078] Lustre: Skipped 1 previous similar message [ 7317.407520] Lustre: server umount lustre-OST0000 complete [ 7319.009145] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7319.012513] LustreError: Skipped 1 previous similar message [ 7319.014384] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7319.020501] Lustre: Skipped 1 previous similar message [ 7319.023358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7321.570043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7321.571267] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7326.689073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7326.692017] Lustre: Skipped 2 previous similar messages [ 7333.344130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7333.417516] Lustre: server umount lustre-MDT0000 complete [ 7334.807173] LustreError: 318443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788755892 with bad export cookie 16825093762909760822 [ 7334.810471] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7334.812937] LustreError: 318443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7334.935339] Lustre: server umount lustre-MDT0001 complete [ 7338.995698] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 7340.163171] Key type lgssc unregistered [ 7361.824366] LNet: 322574:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7361.827320] LNetError: 322574:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7443.751062] LNet: Removed LNI 192.168.206.108@tcp [ 7444.150606] Key type .llcrypt unregistered [ 7444.155597] Key type ._llcrypt unregistered [ 7451.875523] Key type ._llcrypt registered [ 7451.877106] Key type .llcrypt registered [ 7451.921756] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 7459.419917] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 7459.809663] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7459.879117] alg: No test for adler32 (adler32-zlib) [ 7460.762828] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 7460.868179] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 7462.456222] Key type lgssc registered [ 7462.922512] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7475.722761] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 00:40:32 (1788756032) [ 7478.404344] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7478.413545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7479.510723] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7479.541384] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7479.587607] Lustre: lustre-MDT0000: new disk, initializing [ 7479.637130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7479.652533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7481.365265] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7486.338120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7486.395061] Lustre: 325770:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7486.429543] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7486.432416] Lustre: Skipped 1 previous similar message [ 7486.471373] Lustre: lustre-MDT0001: new disk, initializing [ 7486.513919] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7486.530944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7486.535252] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7488.166294] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7492.643618] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7495.040501] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7498.001030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7498.130513] Lustre: lustre-OST0000: new disk, initializing [ 7498.132951] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7498.136282] Lustre: 327738:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7498.184099] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7500.607747] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7504.898485] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7504.903840] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7504.943134] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7504.998525] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7515.106978] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7515.110427] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7515.110480] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7515.115984] Lustre: Skipped 1 previous similar message [ 7517.598779] Lustre: server umount lustre-OST0000 complete [ 7520.226799] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7520.227632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7520.233325] Lustre: Skipped 1 previous similar message [ 7520.239196] Lustre: Skipped 2 previous similar messages [ 7525.349054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7525.352125] Lustre: Skipped 1 previous similar message [ 7525.412238] Lustre: server umount lustre-MDT0000 complete [ 7527.149821] LustreError: 326563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788756084 with bad export cookie 13331066718625933098 [ 7527.153087] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7527.155441] LustreError: 326563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7527.297224] Lustre: server umount lustre-MDT0001 complete [ 7531.649152] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 7532.835846] Key type lgssc unregistered [ 7532.978629] LNet: 329642:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7532.982305] LNetError: 329642:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7532.990446] LNet: Removed LNI 192.168.206.108@tcp [ 7533.380895] Key type .llcrypt unregistered [ 7533.383072] Key type ._llcrypt unregistered [ 7543.502371] Key type ._llcrypt registered [ 7543.504399] Key type .llcrypt registered [ 7543.547783] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 7543.974879] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7544.011286] alg: No test for adler32 (adler32-zlib) [ 7544.905827] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 7545.012926] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 7546.608299] Key type lgssc registered [ 7547.103618] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7550.443987] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7550.455178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7551.646838] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7553.403166] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7557.120821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7557.299095] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7558.970756] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7562.206564] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7564.351467] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7567.289890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7567.449545] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7569.781991] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7572.850339] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7582.797342] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7583.447163] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7587.299334] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7587.301725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7587.304770] Lustre: Skipped 2 previous similar messages [ 7587.310503] Lustre: Skipped 2 previous similar messages [ 7589.457543] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.8@tcp (stopping) [ 7591.335984] Lustre: server umount lustre-MDT0000 complete [ 7592.417641] LustreError: 333525:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7592.425228] LustreError: 333525:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 4 previous similar messages [ 7592.850282] LustreError: 330870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788756150 with bad export cookie 6415470161898624045 [ 7592.853669] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7592.856116] LustreError: 330870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7594.565560] LustreError: 330884:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7594.575234] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.8@tcp (stopping) [ 7597.537353] LustreError: 333805:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7597.539776] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7597.555825] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7599.684386] LustreError: 330884:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7602.657220] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7602.659726] Lustre: Skipped 1 previous similar message [ 7604.806878] LustreError: 330885:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7604.817607] LustreError: 330885:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 7607.264287] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7607.358414] Lustre: server umount lustre-MDT0001 complete [ 7611.047412] Lustre: server umount lustre-OST0000 complete [ 7614.445406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7616.211353] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7617.282163] Lustre: 335068:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.108@tcp [ 7618.074827] Lustre: server umount MGS complete [ 7621.984261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7622.190444] LustreError: 335639:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7622.222462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7624.013696] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7627.756110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7627.900407] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7629.629585] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7633.163376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7635.439528] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7638.050879] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7638.632383] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [ 7640.717863] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7652.800990] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 11 sec [ 7657.955739] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7657.955792] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7657.956420] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7657.956424] Lustre: Skipped 1 previous similar message [ 7657.958303] LustreError: Skipped 1 previous similar message [ 7657.962193] Lustre: Skipped 1 previous similar message [ 7660.047954] Lustre: server umount lustre-OST0000 complete [ 7663.073848] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7663.079356] Lustre: Skipped 1 previous similar message [ 7667.508708] Lustre: server umount lustre-MDT0000 complete [ 7668.193589] LustreError: 335639:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7668.203744] LustreError: 335639:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 7669.006196] LustreError: 336376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788756226 with bad export cookie 6415470161898624878 [ 7669.008811] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7669.011844] LustreError: 336376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7669.139487] Lustre: server umount lustre-MDT0001 complete [ 7673.095287] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 7674.217468] Key type lgssc unregistered [ 7674.349492] LNet: 339094:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7674.354544] LNetError: 339094:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7674.365333] LNet: Removed LNI 192.168.206.108@tcp [ 7674.722124] Key type .llcrypt unregistered [ 7674.723782] Key type ._llcrypt unregistered [ 7681.692738] Key type ._llcrypt registered [ 7681.694584] Key type .llcrypt registered [ 7681.741401] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 7689.076604] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 7689.433314] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7689.450836] alg: No test for adler32 (adler32-zlib) [ 7690.366498] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 7690.512856] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 7692.120199] Key type lgssc registered [ 7692.600291] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7706.804495] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7706.815188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7707.925027] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7707.940671] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7707.986812] Lustre: lustre-MDT0000: new disk, initializing [ 7708.022378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7708.033517] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7709.795360] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7714.815594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7714.865960] Lustre: 342224:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7714.890850] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7714.893875] Lustre: Skipped 1 previous similar message [ 7714.933864] Lustre: lustre-MDT0001: new disk, initializing [ 7714.976365] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7714.993686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7714.999519] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7717.022946] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7722.071405] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7725.329758] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7728.974360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7729.119223] Lustre: lustre-OST0000: new disk, initializing [ 7729.122048] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7729.126101] Lustre: 344190:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7729.168512] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7732.512466] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7735.330873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7735.335188] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7735.395988] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7737.957139] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7739.702503] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7739.793190] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7741.564048] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7741.663981] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7745.509116] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7745.509162] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7745.514381] LustreError: Skipped 1 previous similar message [ 7745.521394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7745.528034] Lustre: Skipped 1 previous similar message [ 7748.641454] Lustre: server umount lustre-OST0000 complete [ 7750.625369] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7750.626204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7750.632086] Lustre: Skipped 2 previous similar messages [ 7755.745211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7755.748009] Lustre: Skipped 2 previous similar messages [ 7756.343565] Lustre: server umount lustre-MDT0000 complete [ 7757.900037] LustreError: 342217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788756315 with bad export cookie 9512402261494195818 [ 7757.900758] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7757.905195] LustreError: 342217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7758.042499] Lustre: server umount lustre-MDT0001 complete [ 7761.880451] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 00:45:18 (1788756318) [ 7769.313540] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 7774.088130] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 7789.899249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7790.026867] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7790.043380] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7790.084764] Lustre: lustre-MDT0000: new disk, initializing [ 7790.129374] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7790.139060] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7791.957534] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7797.279121] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7797.335279] Lustre: 348941:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7797.376876] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7797.377572] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7797.379614] Lustre: Skipped 1 previous similar message [ 7797.422730] Lustre: lustre-MDT0001: new disk, initializing [ 7797.468506] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7797.486972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7797.491157] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7799.316338] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7803.791244] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7806.032320] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7808.679745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7808.797389] Lustre: lustre-OST0000: new disk, initializing [ 7808.799881] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7808.803180] Lustre: 350911:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7808.844844] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7811.147474] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7811.327623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7811.335181] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7811.389665] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7815.591199] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7832.036549] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7832.038077] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7832.038795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7832.045982] LustreError: Skipped 1 previous similar message [ 7832.053071] Lustre: Skipped 1 previous similar message [ 7834.031031] Lustre: server umount lustre-OST0000 complete [ 7837.155305] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7837.162894] Lustre: Skipped 1 previous similar message [ 7841.914728] Lustre: server umount lustre-MDT0000 complete [ 7843.493986] LustreError: 348936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788756401 with bad export cookie 9512402261494196700 [ 7843.499670] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7843.500640] LustreError: 348936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7843.632933] Lustre: server umount lustre-MDT0001 complete [ 7847.879081] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 7849.162314] Key type lgssc unregistered [ 7849.328424] LNet: 353058:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7849.331990] LNetError: 353058:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7849.339355] LNet: Removed LNI 192.168.206.108@tcp [ 7849.727444] Key type .llcrypt unregistered [ 7849.729337] Key type ._llcrypt unregistered [ 7889.574670] Key type ._llcrypt registered [ 7889.576340] Key type .llcrypt registered [ 7889.632626] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 7889.777576] Key type .llcrypt unregistered [ 7889.779310] Key type ._llcrypt unregistered [ 7925.516365] Key type ._llcrypt registered [ 7925.518264] Key type .llcrypt registered [ 7925.566224] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 7925.710340] Key type .llcrypt unregistered [ 7925.712325] Key type ._llcrypt unregistered [ 7939.490787] Key type ._llcrypt registered [ 7939.492058] Key type .llcrypt registered [ 7939.533414] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 7939.658564] Key type .llcrypt unregistered [ 7939.660878] Key type ._llcrypt unregistered [ 7953.150968] Key type ._llcrypt registered [ 7953.152570] Key type .llcrypt registered [ 7953.201606] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 7953.329761] Key type .llcrypt unregistered [ 7953.331418] Key type ._llcrypt unregistered [ 7987.068150] Key type ._llcrypt registered [ 7987.069918] Key type .llcrypt registered [ 7987.113400] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 7987.234371] Key type .llcrypt unregistered [ 7987.236198] Key type ._llcrypt unregistered [ 8000.626673] Key type ._llcrypt registered [ 8000.628180] Key type .llcrypt registered [ 8000.674926] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 8000.808375] Key type .llcrypt unregistered [ 8000.810370] Key type ._llcrypt unregistered [ 8014.816949] Key type ._llcrypt registered [ 8014.818509] Key type .llcrypt registered [ 8014.864775] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 8014.990232] Key type .llcrypt unregistered [ 8014.991845] Key type ._llcrypt unregistered [ 8028.371332] Key type ._llcrypt registered [ 8028.372986] Key type .llcrypt registered [ 8028.434820] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 8028.588416] Key type .llcrypt unregistered [ 8028.591135] Key type ._llcrypt unregistered [ 8041.864436] Key type ._llcrypt registered [ 8041.866090] Key type .llcrypt registered [ 8041.911938] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 8042.051824] Key type .llcrypt unregistered [ 8042.053995] Key type ._llcrypt unregistered [ 8056.021427] Key type ._llcrypt registered [ 8056.023070] Key type .llcrypt registered [ 8056.068568] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 8056.196433] Key type .llcrypt unregistered [ 8056.198318] Key type ._llcrypt unregistered [ 8082.417053] Key type ._llcrypt registered [ 8082.418330] Key type .llcrypt registered [ 8082.466142] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 8082.587417] Key type .llcrypt unregistered [ 8082.590714] Key type ._llcrypt unregistered [ 8096.754839] Key type ._llcrypt registered [ 8096.756092] Key type .llcrypt registered [ 8096.803315] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 8096.935265] Key type .llcrypt unregistered [ 8096.937356] Key type ._llcrypt unregistered [ 8111.111908] Key type ._llcrypt registered [ 8111.113406] Key type .llcrypt registered [ 8111.166928] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 8111.305385] Key type .llcrypt unregistered [ 8111.307538] Key type ._llcrypt unregistered [ 8135.209452] Key type ._llcrypt registered [ 8135.212059] Key type .llcrypt registered [ 8135.260669] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 8142.734459] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 8143.183706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8143.237156] alg: No test for adler32 (adler32-zlib) [ 8144.153160] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 8144.266152] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 8145.864225] Key type lgssc registered [ 8146.335884] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8163.358842] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8163.377787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8164.482674] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [ 8164.498569] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [ 8164.543837] Lustre: tst250fs-MDT0000: new disk, initializing [ 8164.596635] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8164.610061] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8166.689228] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8171.991858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8172.060128] Lustre: 372912:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8172.098514] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [ 8172.102430] Lustre: Skipped 1 previous similar message [ 8172.142838] Lustre: tst250fs-MDT0001: new disk, initializing [ 8172.190313] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8172.205531] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8172.211790] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8173.939052] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8176.702905] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8179.646369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8179.767085] Lustre: tst250fs-OST0000: new disk, initializing [ 8179.770413] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [ 8179.774361] Lustre: 374547:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8179.814316] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8182.220699] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8185.350439] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8185.356423] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8185.420159] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8187.111180] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8187.181421] Lustre: tst250fs-OST0001: new disk, initializing [ 8187.183603] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [ 8187.186637] Lustre: 375418:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8187.230940] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8189.709884] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8192.706464] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [ 8192.859488] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8192.863583] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8192.918480] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8208.355105] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8208.356145] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8208.357429] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 8208.368627] Lustre: Skipped 1 previous similar message [ 8210.534328] Lustre: server umount tst250fs-MDT0000 complete [ 8211.995083] LustreError: 372905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788756769 with bad export cookie 8577842793965070857 [ 8211.998485] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8212.000011] LustreError: 372905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8212.129984] Lustre: server umount tst250fs-MDT0001 complete [ 8223.825260] Lustre: server umount tst250fs-OST0000 complete [ 8235.693139] Lustre: server umount tst250fs-OST0001 complete [ 8243.248307] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 8247.297396] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 8262.552247] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 00:53:39 (1788756819) [ 8263.316060] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [ 8270.296192] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 8274.318655] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 8289.351575] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8028 sec ========== 00:54:06 (1788756846) [ 8290.252785] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 00:54:06 (1788756846) === [ 8292.037493] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 00:54:08 (1788756848) === [ 8309.863194] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 8311.207543] Key type lgssc unregistered [ 8311.375377] LNet: 384538:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8311.379708] LNetError: 384538:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8311.392694] LNet: Removed LNI 192.168.206.108@tcp [ 8311.897403] Key type .llcrypt unregistered [ 8311.900539] Key type ._llcrypt unregistered