[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 496476255 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2640MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002378] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.007426] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009011] pid_max: default: 32768 minimum: 301 [ 0.010131] LSM: Security Framework initializing [ 0.012017] Yama: becoming mindful. [ 0.013033] SELinux: Initializing. [ 0.014056] *** VALIDATE selinux *** [ 0.022404] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026661] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027135] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028114] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029106] *** VALIDATE tmpfs *** [ 0.030430] *** VALIDATE proc *** [ 0.031252] *** VALIDATE cgroup *** [ 0.032010] *** VALIDATE cgroup2 *** [ 0.034263] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035155] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037023] Spectre V2 : User space: Vulnerable [ 0.039003] Speculative Store Bypass: Vulnerable [ 0.041965] debug: unmapping init [mem 0xffffffff87c59000-0xffffffff87c60fff] [ 0.043183] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044590] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045020] ... version: 2 [ 0.046012] ... bit width: 48 [ 0.047011] ... generic registers: 4 [ 0.048014] ... value mask: 0000ffffffffffff [ 0.049015] ... max period: 00007fffffffffff [ 0.050013] ... fixed-purpose events: 3 [ 0.051014] ... event mask: 000000070000000f [ 0.052289] rcu: Hierarchical SRCU implementation. [ 0.054375] smp: Bringing up secondary CPUs ... [ 0.055574] x86: Booting SMP configuration: [ 0.056023] .... node #0, CPUs: #1 #2 #3 [ 0.060175] smp: Brought up 1 node, 4 CPUs [ 0.062018] smpboot: Max logical packages: 1 [ 0.063014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.227151] node 0 deferred pages initialised in 162ms [ 0.230469] devtmpfs: initialized [ 0.231245] x86/mm: Memory block size: 128MB [ 0.234978] gcov: version magic: 0x41383552 [ 0.237307] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.241101] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.244363] pinctrl core: initialized pinctrl subsystem [ 0.246258] [ 0.246841] ************************************************************* [ 0.249013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.252013] ** ** [ 0.254013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.257014] ** ** [ 0.260014] ** This means that this kernel is built to expose internal ** [ 0.262012] ** IOMMU data structures, which may compromise security on ** [ 0.265013] ** your system. ** [ 0.267011] ** ** [ 0.270015] ** If you see this message and you are not debugging the ** [ 0.273013] ** kernel, report this immediately to your vendor! ** [ 0.275012] ** ** [ 0.278015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.280011] ************************************************************* [ 0.282627] NET: Registered protocol family 16 [ 0.285403] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.288058] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.291125] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.295027] cpuidle: using governor menu [ 0.296702] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.298620] PCI: Using configuration type 1 for base access [ 0.299000] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.309146] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.312031] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.315232] cryptd: max_cpu_qlen set to 1000 [ 0.319374] ACPI: Added _OSI(Module Device) [ 0.320020] ACPI: Added _OSI(Processor Device) [ 0.321000] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.322013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.327828] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.335335] ACPI: Interpreter enabled [ 0.337065] ACPI: PM: (supports S0 S3 S4 S5) [ 0.339012] ACPI: Using IOAPIC for interrupt routing [ 0.341117] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.345514] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.357209] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.360047] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.363019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.366190] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.371544] acpiphp: Slot [2] registered [ 0.374180] acpiphp: Slot [5] registered [ 0.376232] acpiphp: Slot [6] registered [ 0.377206] acpiphp: Slot [7] registered [ 0.379137] acpiphp: Slot [8] registered [ 0.381129] acpiphp: Slot [9] registered [ 0.382128] acpiphp: Slot [10] registered [ 0.384113] acpiphp: Slot [3] registered [ 0.385093] acpiphp: Slot [4] registered [ 0.387106] acpiphp: Slot [11] registered [ 0.388106] acpiphp: Slot [12] registered [ 0.389074] acpiphp: Slot [13] registered [ 0.390085] acpiphp: Slot [14] registered [ 0.391112] acpiphp: Slot [15] registered [ 0.393107] acpiphp: Slot [16] registered [ 0.394100] acpiphp: Slot [17] registered [ 0.396124] acpiphp: Slot [18] registered [ 0.397087] acpiphp: Slot [19] registered [ 0.398079] acpiphp: Slot [20] registered [ 0.400096] acpiphp: Slot [21] registered [ 0.402099] acpiphp: Slot [22] registered [ 0.403101] acpiphp: Slot [23] registered [ 0.405102] acpiphp: Slot [24] registered [ 0.406116] acpiphp: Slot [25] registered [ 0.408102] acpiphp: Slot [26] registered [ 0.410110] acpiphp: Slot [27] registered [ 0.411100] acpiphp: Slot [28] registered [ 0.413109] acpiphp: Slot [29] registered [ 0.415131] acpiphp: Slot [30] registered [ 0.417133] acpiphp: Slot [31] registered [ 0.418072] PCI host bridge to bus 0000:00 [ 0.419016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.421031] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.425025] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.427015] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.429017] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.431020] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.432194] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.435412] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.438431] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.448016] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.455657] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.458018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.460014] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.462014] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.466532] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.469895] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.473047] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.477169] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.484014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.500014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.506014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.513228] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.521000] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.531013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.551015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.561700] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.572014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.592014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.618015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.632044] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.647013] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.658016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.685015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.698712] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.709014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.718014] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.742013] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.754625] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.763014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.769015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.790015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.800590] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.811014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.817013] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.845020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.857181] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.859379] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.861368] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.863532] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.867223] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.874032] iommu: Default domain type: Passthrough [ 0.875476] SCSI subsystem initialized [ 0.877139] ACPI: bus type USB registered [ 0.878181] usbcore: registered new interface driver usbfs [ 0.881096] usbcore: registered new interface driver hub [ 0.883089] usbcore: registered new device driver usb [ 0.884162] pps_core: LinuxPPS API ver. 1 registered [ 0.886011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.890043] PTP clock support registered [ 0.891116] EDAC MC: Ver: 3.0.0 [ 0.893128] PCI: Using ACPI for IRQ routing [ 0.894745] NetLabel: Initializing [ 0.896009] NetLabel: domain hash size = 128 [ 0.897007] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.898070] NetLabel: unlabeled traffic allowed by default [ 0.901072] vgaarb: loaded [ 0.902244] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.903012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.908294] clocksource: Switched to clocksource kvm-clock [ 1.017293] VFS: Disk quotas dquot_6.6.0 [ 1.018695] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.021503] *** VALIDATE ramfs *** [ 1.023352] *** VALIDATE hugetlbfs *** [ 1.025915] pnp: PnP ACPI init [ 1.028948] pnp: PnP ACPI: found 6 devices [ 1.060226] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.063988] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.068384] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.072578] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.077314] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.081153] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.085767] NET: Registered protocol family 2 [ 1.089122] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.094558] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.098357] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.104754] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.109820] TCP: Hash tables configured (established 65536 bind 65536) [ 1.113761] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.118363] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.120906] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.123801] NET: Registered protocol family 1 [ 1.126694] RPC: Registered named UNIX socket transport module. [ 1.128699] RPC: Registered udp transport module. [ 1.130517] RPC: Registered tcp transport module. [ 1.131921] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.133814] NET: Registered protocol family 44 [ 1.135199] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.137043] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.139568] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.142688] PCI: CLS 0 bytes, default 64 [ 1.144298] Unpacking initramfs... [ 2.569455] debug: unmapping init [mem 0xffff93e73cc54000-0xffff93e73ffbffff] [ 2.573437] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.575644] software IO TLB: mapped [mem 0x00000000a1000000-0x00000000a5000000] (64MB) [ 2.578426] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.084232] Initialise system trusted keyrings [ 3.086110] Key type blacklist registered [ 3.088060] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.099451] zbud: loaded [ 3.103081] *** VALIDATE nfs *** [ 3.104481] *** VALIDATE nfs4 *** [ 3.106378] pstore: using deflate compression [ 3.110020] Platform Keyring initialized [ 3.208978] NET: Registered protocol family 38 [ 3.211380] Key type asymmetric registered [ 3.213148] Asymmetric key parser 'x509' registered [ 3.215417] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.220101] io scheduler mq-deadline registered [ 3.224406] io scheduler kyber registered [ 3.229943] io scheduler bfq registered [ 3.233076] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.238891] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.241853] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.245110] ACPI: Power Button [PWRF] [ 3.251119] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.258482] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.283855] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.302683] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.332183] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.358993] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.385960] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.391555] Non-volatile memory driver v1.3 [ 3.394379] Linux agpgart interface v0.103 [ 3.424981] virtio_blk virtio1: [vda] 149784 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.427870] vda: detected capacity change from 0 to 76689408 [ 3.451797] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.455452] vdb: detected capacity change from 0 to 1073741824 [ 3.476443] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.480156] vdc: detected capacity change from 0 to 2621440000 [ 3.495285] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.498735] vdd: detected capacity change from 0 to 2621440000 [ 3.522297] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.525429] vde: detected capacity change from 0 to 4294967296 [ 3.548915] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.552636] vdf: detected capacity change from 0 to 4294967296 [ 3.563675] libphy: Fixed MDIO Bus: probed [ 3.570579] usbcore: registered new interface driver usbserial_generic [ 3.573800] usbserial: USB Serial support registered for generic [ 3.576728] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.582396] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.584288] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.586761] mousedev: PS/2 mouse device common for all mice [ 3.590261] rtc_cmos 00:05: RTC can wake from S4 [ 3.594688] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.600874] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.603630] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.606900] rtc_cmos 00:05: registered as rtc0 [ 3.612123] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.615185] intel_pstate: CPU model not supported [ 3.618517] hid: raw HID events driver (C) Jiri Kosina [ 3.621223] usbcore: registered new interface driver usbhid [ 3.623784] usbhid: USB HID core driver [ 3.625941] drop_monitor: Initializing network drop monitor service [ 3.629258] Initializing XFRM netlink socket [ 3.631286] NET: Registered protocol family 10 [ 3.634652] Segment Routing with IPv6 [ 3.636185] NET: Registered protocol family 17 [ 3.639040] mpls_gso: MPLS GSO support [ 3.643694] RAS: Correctable Errors collector initialized. [ 3.646051] AVX version of gcm_enc/dec engaged. [ 3.647287] AES CTR mode by8 optimization enabled [ 3.729033] sched_clock: Marking stable (3729007714, 0)->(4627631647, -898623933) [ 3.732497] registered taskstats version 1 [ 3.734566] Loading compiled-in X.509 certificates [ 3.737918] zswap: loaded using pool lzo/zbud [ 3.762933] Key type big_key registered [ 3.778899] Key type encrypted registered [ 3.781157] ima: No TPM chip found, activating TPM-bypass! [ 3.783755] ima: Allocated hash algorithm: sha1 [ 3.786655] ima: No architecture policies found [ 3.788372] evm: Initialising EVM extended attributes: [ 3.790531] evm: security.selinux [ 3.791856] evm: security.ima [ 3.793240] evm: security.capability [ 3.794954] evm: HMAC attrs: 0x1 [ 3.797611] rtc_cmos 00:05: setting system clock to 2026-09-07 16:52:48 UTC (1788799968) [ 3.804533] debug: unmapping init [mem 0xffffffff88c03000-0xffffffff88dfffff] [ 3.807905] debug: unmapping init [mem 0xffffffff87982000-0xffffffff87c58fff] [ 3.817088] Write protecting the kernel read-only data: 28672k [ 3.820356] debug: unmapping init [mem 0xffffffff86003000-0xffffffff861fffff] [ 3.823128] debug: unmapping init [mem 0xffffffff86914000-0xffffffff869fffff] [ 3.860427] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.869370] systemd[1]: Detected virtualization kvm. [ 3.871368] systemd[1]: Detected architecture x86-64. [ 3.873810] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.904468] systemd[1]: No hostname configured. [ 3.906590] systemd[1]: Set hostname to . [ 3.908960] random: systemd: uninitialized urandom read (16 bytes read) [ 3.911685] systemd[1]: Initializing machine ID from random generator. [ 3.974423] random: ln: uninitialized urandom read (6 bytes read) [ 4.073855] random: systemd: uninitialized urandom read (16 bytes read) [ 4.077195] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 4.082699] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 4.087493] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.799631] device-mapper: uevent: version 1.0.3 [ 4.802395] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.564409] virtio_net virtio0 ens2: renamed from eth0 [ 5.566621] random: fast init done [ 5.725568] scsi host0: ata_piix [ 5.745803] scsi host1: ata_piix [ 5.747988] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.751017] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.293990] dracut-initqueue[586]: RTNETLINK answers: File exists [ 10.300879] random: crng init done [ 10.302179] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.821820] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.044307] printk: systemd: 25 output lines suppressed due to ratelimiting [ 12.372739] SELinux: Disabled at runtime. [ 12.436245] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.443483] systemd[1]: Detected virtualization kvm. [ 12.445481] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.981645] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.985154] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.006518] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.013932] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.017899] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.026673] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.034752] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on RPCbind Server Activation Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... [ 13.117855] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Huge Pages File System... [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Apply Kernel Variables... Mounting Kernel Debug File System... [ OK ] Listening on Process Core Dump Socket. Starting udev Coldplug all Devices... Mounting POSIX Message Queue File System... [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.502698] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.761619] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.790858] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.933727] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.956279] EDAC sbridge: Ver: 1.1.2 [ 15.766115] Key type dns_resolver registered [ 16.071667] NFS: Registering the id_resolver key type [ 16.074721] Key type id_resolver registered [ 16.076353] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Load/Save Random Seed. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Login Service. [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg656-server login: [ 69.226104] hrtimer: interrupt took 3636051 ns [ 69.739752] libcfs: loading out-of-tree module taints kernel. [ 69.813248] Key type ._llcrypt registered [ 69.826252] Key type .llcrypt registered [ 70.002184] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_hostid [ 90.617310] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing load_modules_local [ 92.672064] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 92.695990] alg: No test for adler32 (adler32-zlib) [ 94.454437] Lustre: Lustre: Build Version: 2.17.58_2_gefa5002 [ 95.815032] LNet: Added LNI 192.168.206.156@tcp [8/256/0/180] [ 97.866291] Key type lgssc registered [ 100.386397] Lustre: Echo OBD driver; http://www.lustre.org/ [ 120.944589] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 163.025367] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing load_modules_local [ 176.084355] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 176.148029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 177.410917] Lustre: First target registration detected: opening a one-time 24-hour registration window. Use 'lctl set_param allow_register=0' to close it early, or 'lctl set_param -P allow_register=0' to lock down permanently. [ 177.461079] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 177.508426] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 177.664726] Lustre: lustre-MDT0000: new disk, initializing [ 177.786983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 177.807194] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 182.513692] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 196.057932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 196.179766] Lustre: 6514:0:(mgs_llog.c:1462:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 196.263471] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 196.272508] Lustre: Skipped 1 previous similar message [ 196.305391] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 196.393949] Lustre: lustre-MDT0001: new disk, initializing [ 196.435108] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 196.461434] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 196.473484] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 200.531286] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 205.476718] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 215.769334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 216.062923] Lustre: lustre-OST0000: new disk, initializing [ 216.067331] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 216.072658] Lustre: 8452:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 216.159761] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 221.737620] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 221.744448] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 221.840740] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 222.726724] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 236.822880] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 236.955653] Lustre: lustre-OST0001: new disk, initializing [ 236.959303] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 236.964696] Lustre: 9526:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 237.054562] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 242.568443] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 245.778096] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 245.791568] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 245.831509] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 255.310596] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 260.611153] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 266.796369] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing check_logdir /tmp/testlogs/ [ 271.626833] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing yml_node [ 275.475286] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 278.271441] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 281.219485] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 283.028228] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Sep 7 12:57:26 EDT 2026 [ 299.963671] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 301.354862] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 317.410024] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 317.412134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 317.426773] Lustre: Skipped 1 previous similar message [ 317.441508] Lustre: Skipped 3 previous similar messages [ 322.503034] Lustre: server umount lustre-MDT0000 complete [ 325.602205] LustreError: 9525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788800290 with bad export cookie 10476266364675368004 [ 325.607206] LustreError: MGC192.168.206.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 325.613280] LustreError: 9525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 327.649404] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 327.651503] LustreError: 6521:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 327.671307] Lustre: Skipped 2 previous similar messages [ 327.672464] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 327.694989] LustreError: 6521:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 327.716702] Lustre: Skipped 1 previous similar message [ 330.657802] LustreError: 6522:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 330.657898] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 330.689632] Lustre: Skipped 1 previous similar message [ 332.017475] Lustre: server umount lustre-MDT0001 complete [ 335.931422] Lustre: server umount lustre-OST0000 complete [ 339.119831] Lustre: server umount lustre-OST0001 complete [ 344.897913] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_hostid [ 353.082775] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing load_modules_local [ 383.803961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 384.056779] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 384.094278] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 384.206109] Lustre: lustre-MDT0000: new disk, initializing [ 384.305791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 384.319578] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 388.163754] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 397.429227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 397.523785] Lustre: 15073:0:(mgs_llog.c:1462:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 397.571697] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 397.575163] Lustre: Skipped 1 previous similar message [ 397.641434] Lustre: lustre-MDT0001: new disk, initializing [ 397.707612] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 397.758378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 397.772713] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 401.414427] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 409.239307] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 414.259867] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 419.730187] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 419.976909] Lustre: lustre-OST0000: new disk, initializing [ 419.982293] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 419.989919] Lustre: 17041:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 420.086575] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 421.134680] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 421.145143] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 421.231197] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 425.565351] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 433.482560] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 437.879188] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 438.081661] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 442.636703] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 442.816793] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 446.944927] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 446.945985] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 446.961296] LustreError: Skipped 1 previous similar message [ 446.975072] Lustre: Skipped 1 previous similar message [ 446.980391] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 446.993691] Lustre: Skipped 1 previous similar message [ 450.687321] Lustre: server umount lustre-OST0000 complete [ 454.112438] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 454.122547] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 454.146257] Lustre: Skipped 1 previous similar message [ 457.184973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 457.187772] Lustre: Skipped 2 previous similar messages [ 460.387378] Lustre: server umount lustre-MDT0000 complete [ 462.304037] LustreError: 15085:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 462.332702] LustreError: 15085:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 464.395740] LustreError: 15065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788800429 with bad export cookie 10476266364675369537 [ 464.407029] LustreError: 15065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 464.417083] LustreError: MGC192.168.206.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 464.863063] Lustre: server umount lustre-MDT0001 complete [ 470.847920] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 13:00:34 (1788800434) [ 488.347679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 488.951181] LustreError: 19914:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 488.985933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 492.824115] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 494.052079] LustreError: 19915:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 501.281344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 505.294170] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 512.781351] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 518.017197] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 519.805235] Lustre: Failing over lustre-MDT0000 [ 522.053026] Lustre: server umount lustre-MDT0000 complete [ 522.216373] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 522.230443] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 522.258746] Lustre: Skipped 1 previous similar message [ 522.270217] LustreError: 20656:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 522.280845] LustreError: 20656:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 539.615627] Lustre: 3649:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788800488/real 1788800488] req@ffff93e7b2175180 x1875692811866752/t0(0) o400->MGC192.168.206.156@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788800504 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 539.635272] LustreError: MGC192.168.206.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 539.657280] LustreError: 19915:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 539.675468] LustreError: 19915:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 541.408302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 554.713450] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 555.500439] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 555.519954] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 555.572397] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 563.975957] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 569.826518] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 569.830194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 569.844263] Lustre: Skipped 2 previous similar messages [ 573.899651] Lustre: server umount lustre-MDT0000 complete [ 576.993081] LustreError: 20656:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 577.013980] LustreError: 20656:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 577.542949] LustreError: 20663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788800542 with bad export cookie 10476266364675370951 [ 577.545076] LustreError: MGC192.168.206.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 577.553761] LustreError: 20663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 577.840376] Lustre: server umount lustre-MDT0001 complete [ 584.915409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 585.270573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 585.275579] Lustre: Skipped 2 previous similar messages [ 589.329219] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 597.453406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 602.578588] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 611.271757] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 617.254607] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 619.747753] Lustre: Failing over lustre-MDT0000 [ 620.249245] Lustre: server umount lustre-MDT0000 complete [ 622.818387] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 622.827966] Lustre: Skipped 1 previous similar message [ 639.456550] Lustre: 3649:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788800587/real 1788800587] req@ffff93e7a657dc00 x1875692811896832/t0(0) o400->MGC192.168.206.156@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788800603 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 639.507149] LustreError: MGC192.168.206.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 639.558552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 642.592635] LustreError: 24656:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 642.605112] LustreError: 24656:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 651.163319] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 656.366671] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 656.384849] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 656.386678] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 656.394722] Lustre: Skipped 1 previous similar message [ 657.221089] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 667.515951] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 671.713121] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 671.734329] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 671.764788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 671.773062] Lustre: Skipped 3 previous similar messages [ 677.334705] Lustre: server umount lustre-MDT0000 complete [ 681.391328] LustreError: 23246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788800646 with bad export cookie 10476266364675371889 [ 681.393777] LustreError: MGC192.168.206.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 681.398132] LustreError: 23246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 681.699074] Lustre: server umount lustre-MDT0001 complete [ 689.961172] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 13:04:13 (1788800653) [ 705.928769] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_hostid [ 715.473311] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing load_modules_local [ 755.918680] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing load_modules_local [ 768.457231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 768.741525] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 768.807760] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 768.919623] Lustre: lustre-MDT0000: new disk, initializing [ 769.013787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 769.020574] Lustre: Skipped 2 previous similar messages [ 769.042508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 773.944300] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 784.726345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 784.842497] Lustre: 29642:0:(mgs_llog.c:1462:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 785.188801] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 789.679993] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 799.304406] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 804.984892] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 812.562138] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 812.792246] Lustre: 31610:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 814.268280] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 814.308243] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 814.362970] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 814.374634] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 818.996802] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 828.571395] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 834.230785] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 860.642307] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 860.644194] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 860.649925] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 860.649932] Lustre: Skipped 2 previous similar messages [ 860.702578] Lustre: Skipped 2 previous similar messages [ 870.367161] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 870.601617] Lustre: server umount lustre-OST0000 complete [ 880.669384] Lustre: server umount lustre-MDT0000 complete [ 881.121764] LustreError: 29654:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 881.145431] LustreError: 29654:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 884.650146] LustreError: 31621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788800849 with bad export cookie 10476266364675372295 [ 884.652228] LustreError: MGC192.168.206.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 884.659703] LustreError: 31621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 884.988862] Lustre: server umount lustre-MDT0001 complete [ 898.440880] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing unload_modules_local [ 901.900885] Key type lgssc unregistered [ 902.139800] LNet: 33586:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 902.147555] LNetError: 33586:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 902.189134] LNet: Removed LNI 192.168.206.156@tcp [ 903.555324] Key type .llcrypt unregistered [ 903.558757] Key type ._llcrypt unregistered [ 926.980721] Key type ._llcrypt registered [ 926.983081] Key type .llcrypt registered [ 927.082750] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing load_modules_local [ 927.862551] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 927.919074] alg: No test for adler32 (adler32-zlib) [ 929.275804] Lustre: Lustre: Build Version: 2.17.58_2_gefa5002 [ 929.598586] LNet: Added LNI 192.168.206.156@tcp [8/256/0/180] [ 931.335858] Key type lgssc registered [ 932.647764] Lustre: Echo OBD driver; http://www.lustre.org/ [ 942.428724] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 942.473047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 943.922232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 950.395669] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 953.320215] LustreError: 34997:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 953.353808] LustreError: 34997:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 958.437817] LustreError: 34998:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 959.349548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 959.733584] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 964.883166] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 971.942850] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 977.755672] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 984.953680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 985.324656] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 988.444327] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 991.989722] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 999.507139] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1004.785156] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1015.319569] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 13:09:38 (1788800978) [ 1016.872089] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1018.440446] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1025.268871] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 13:09:48 (1788800988) [ 1033.679390] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 13:09:56 (1788800996) [ 1034.887330] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1043.729620] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1043.734900] Lustre: Skipped 2 previous similar messages [ 1045.230298] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 13:10:08 (1788801008) [ 1048.811886] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1048.816153] Lustre: Skipped 3 previous similar messages [ 1056.867171] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1056.871975] Lustre: Skipped 3 previous similar messages [ 1071.530873] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 13:10:34 (1788801034) [ 1074.629180] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1078.175798] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1087.839498] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 13:10:50 (1788801050) [ 1089.531317] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1089.536819] Lustre: Skipped 6 previous similar messages [ 1102.767614] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 13:11:05 (1788801065) [ 1117.676440] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1117.681205] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1117.703662] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1117.709123] Lustre: Skipped 1 previous similar message [ 1122.788708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1122.801063] Lustre: Skipped 1 previous similar message [ 1123.507886] Lustre: server umount lustre-OST0000 complete [ 1127.907043] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1127.914813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1127.926379] Lustre: Skipped 2 previous similar messages [ 1127.938731] Lustre: Skipped 1 previous similar message [ 1133.025205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1133.033762] Lustre: Skipped 1 previous similar message [ 1133.459989] Lustre: server umount lustre-MDT0000 complete [ 1137.203082] LustreError: 39424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788801101 with bad export cookie 2593526761234623500 [ 1137.206900] LustreError: MGC192.168.206.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1137.211417] LustreError: 39424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1137.778558] Lustre: server umount lustre-MDT0001 complete [ 1147.760594] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing unload_modules_local [ 1150.560623] Key type lgssc unregistered [ 1150.926796] LNet: 42853:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1150.945456] LNetError: 42853:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1151.987173] LNet: Removed LNI 192.168.206.156@tcp [ 1152.997879] Key type .llcrypt unregistered [ 1152.999603] Key type ._llcrypt unregistered [ 1170.176689] Key type ._llcrypt registered [ 1170.178994] Key type .llcrypt registered [ 1170.269612] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_hostid [ 1184.148983] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing load_modules_local [ 1185.028032] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1185.131701] alg: No test for adler32 (adler32-zlib) [ 1186.209056] Lustre: Lustre: Build Version: 2.17.58_2_gefa5002 [ 1186.457651] LNet: Added LNI 192.168.206.156@tcp [8/256/0/180] [ 1188.097253] Key type lgssc registered [ 1189.144723] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1225.870111] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1225.897734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1227.210651] Lustre: First target registration detected: opening a one-time 24-hour registration window. Use 'lctl set_param allow_register=0' to close it early, or 'lctl set_param -P allow_register=0' to lock down permanently. [ 1227.234769] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1227.281064] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1227.382549] Lustre: lustre-MDT0000: new disk, initializing [ 1227.454208] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1227.470783] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1232.021310] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 1242.100154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1242.226978] Lustre: 46004:0:(mgs_llog.c:1462:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1242.287818] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1242.292435] Lustre: Skipped 1 previous similar message [ 1242.362477] Lustre: lustre-MDT0001: new disk, initializing [ 1242.453280] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1242.489202] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1242.512228] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1246.818213] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 1255.023350] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1260.022536] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1266.345702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1266.655636] Lustre: lustre-OST0000: new disk, initializing [ 1266.658505] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1266.666572] Lustre: 47970:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1266.753927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1272.318286] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 1274.428107] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1274.438721] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1274.516548] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1280.843953] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1284.468264] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1284.625791] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1288.173276] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1288.311555] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1290.219669] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1290.223217] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1290.225353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1290.231642] LustreError: Skipped 1 previous similar message [ 1290.264660] Lustre: Skipped 1 previous similar message [ 1295.332235] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1295.342843] Lustre: Skipped 2 previous similar messages [ 1300.454916] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1300.463657] Lustre: Skipped 1 previous similar message [ 1304.031118] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1304.180311] Lustre: server umount lustre-OST0000 complete [ 1309.152590] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1309.163849] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1309.179959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1310.690098] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1313.876172] Lustre: server umount lustre-MDT0000 complete [ 1317.197508] LustreError: 45996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788801281 with bad export cookie 17601184169648631047 [ 1317.203880] LustreError: MGC192.168.206.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1317.210340] LustreError: 45996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1317.690384] Lustre: server umount lustre-MDT0001 complete [ 1326.451732] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 13:14:49 (1788801289) [ 1332.056354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1332.527300] LustreError: 50433:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1332.541740] LustreError: 50433:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1332.617540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1336.463442] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 1337.825840] LustreError: 50434:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1342.954293] LustreError: 50433:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1344.632792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1344.890408] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1348.699572] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 1355.620930] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1360.798629] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1366.843964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1367.217354] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1373.018956] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 1380.189348] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1384.342792] LustreError: 52962:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1385.098852] Lustre: Modifying parameter general.timeout=1 in log params [ 1386.726161] Lustre: Modifying parameter general.timeout=3 in log params [ 1386.729064] Lustre: Skipped 1 previous similar message [ 1389.149856] Lustre: Modifying parameter general.timeout=6 in log params [ 1389.155515] Lustre: Skipped 2 previous similar messages [ 1393.721760] Lustre: Modifying parameter general.timeout=12 in log params [ 1393.725361] Lustre: Skipped 5 previous similar messages [ 1401.840501] Lustre: Modifying parameter general.timeout=22 in log params [ 1401.850103] Lustre: Skipped 9 previous similar messages [ 1418.168796] Lustre: Modifying parameter general.timeout=43 in log params [ 1418.171328] Lustre: Skipped 20 previous similar messages [ 1450.221303] Lustre: Modifying parameter general.timeout=84 in log params [ 1450.224670] Lustre: Skipped 40 previous similar messages [ 1493.529629] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 13:17:36 (1788801456) [ 1496.003238] Lustre: 59894:0:(mgs_llog.c:1462:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1497.554700] Lustre: 60004:0:(mgs_llog.c:1462:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1501.290187] Lustre: 60201:0:(mgs_llog.c:1462:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1501.305962] Lustre: 60201:0:(mgs_llog.c:1462:mgs_modify_param()) Skipped 1 previous similar message [ 1505.354348] Lustre: 60415:0:(mgs_llog.c:1462:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1505.362782] Lustre: 60415:0:(mgs_llog.c:1462:mgs_modify_param()) Skipped 1 previous similar message [ 1515.739681] Lustre: Disabling parameter general.ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=50 in log params [ 1515.751474] Lustre: Skipped 60 previous similar messages [ 1530.745709] Lustre: 61770:0:(mgs_llog.c:1462:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1530.751561] LustreError: 61770:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1534.362850] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 13:18:17 (1788801497) [ 1544.682811] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1544.691157] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1544.709065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1544.715697] Lustre: Skipped 2 previous similar messages [ 1549.284900] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1549.305020] Lustre: Skipped 1 previous similar message [ 1550.721375] Lustre: server umount lustre-MDT0000 complete [ 1554.403899] LustreError: 50433:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1554.424116] LustreError: 50433:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1554.553602] LustreError: 53555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788801519 with bad export cookie 17601184169648631929 [ 1554.571303] LustreError: MGC192.168.206.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1554.573140] LustreError: 53555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 1555.064867] Lustre: server umount lustre-MDT0001 complete [ 1569.881909] Lustre: server umount lustre-OST0000 complete [ 1597.723748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1597.958601] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1598.129317] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1602.145112] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 1610.878718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1611.120135] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1611.181258] Lustre: 64306:0:(mgs_llog.c:1462:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1616.829507] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 1624.159026] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1630.391227] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1636.614925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1636.816582] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1637.098068] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1637.103751] Lustre: Skipped 1 previous similar message [ 1642.902961] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 1649.836797] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1651.680162] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1651.687356] Lustre: Skipped 8 previous similar messages [ 1651.727445] Lustre: 66881:0:(mgs_llog.c:1462:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1659.366017] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1659.382987] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1659.404882] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1659.411767] Lustre: Skipped 4 previous similar messages [ 1664.866639] Lustre: server umount lustre-OST0000 complete [ 1675.450429] Lustre: server umount lustre-MDT0000 complete [ 1679.840300] LustreError: 65437:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1679.863240] LustreError: 65437:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1680.604944] LustreError: 65061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788801645 with bad export cookie 17601184169648634841 [ 1680.618414] LustreError: MGC192.168.206.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1680.618808] LustreError: 65061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1680.922862] Lustre: server umount lustre-MDT0001 complete [ 1694.210942] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing unload_modules_local [ 1698.438537] Key type lgssc unregistered [ 1698.833073] LNet: 68090:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1698.846583] LNetError: 68090:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1698.866685] LNet: Removed LNI 192.168.206.156@tcp [ 1699.692075] Key type .llcrypt unregistered [ 1699.695397] Key type ._llcrypt unregistered [ 1725.345983] Key type ._llcrypt registered [ 1725.348509] Key type .llcrypt registered [ 1725.427170] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing load_modules_local [ 1726.649882] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1726.702828] alg: No test for adler32 (adler32-zlib) [ 1727.710989] Lustre: Lustre: Build Version: 2.17.58_2_gefa5002 [ 1727.993096] LNet: Added LNI 192.168.206.156@tcp [8/256/0/180] [ 1729.671437] Key type lgssc registered [ 1730.686823] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1737.797310] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1737.823715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1739.191711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1743.621468] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 1748.452289] LustreError: 69502:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1748.479761] LustreError: 69502:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1751.298353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1751.884468] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1757.624422] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 1765.630487] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1770.759823] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1776.402259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1776.614223] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 1776.922495] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1782.215541] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 1789.702272] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1797.444166] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1807.850216] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1807.862914] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1807.888981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1809.893221] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1809.895784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1809.933917] Lustre: Skipped 1 previous similar message [ 1809.958853] Lustre: Skipped 2 previous similar messages [ 1812.860088] LustreError: 72542:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (00000000b8e87dd7) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 1813.089306] Lustre: server umount lustre-MDT0000 complete [ 1815.016515] LustreError: 69501:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1815.036042] LustreError: 69501:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1817.735608] LustreError: 69482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788801782 with bad export cookie 9045482694168622793 [ 1817.749465] LustreError: MGC192.168.206.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1818.349580] Lustre: server umount lustre-MDT0001 complete [ 1833.933978] Lustre: server umount lustre-OST0000 complete [ 1863.466451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1864.155209] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1864.173464] Lustre: First target registration detected: opening a one-time 24-hour registration window. Use 'lctl set_param allow_register=0' to close it early, or 'lctl set_param -P allow_register=0' to lock down permanently. [ 1864.241040] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1864.558862] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1871.520549] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 1883.824705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1883.915816] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1883.939915] Lustre: 74379:0:(mgs_llog.c:1462:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1884.352336] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1889.160614] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 1895.188898] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1900.973506] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1907.875096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1908.104833] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1908.543860] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1914.737635] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 1922.661785] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1924.929659] Lustre: 76958:0:(mgs_llog.c:1462:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1926.774130] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1935.345651] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1935.351048] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1935.370916] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1938.555394] Lustre: server umount lustre-OST0000 complete [ 1945.568809] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1945.575038] LustreError: Skipped 1 previous similar message [ 1945.578880] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1945.590785] Lustre: Skipped 1 previous similar message [ 1945.596561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1945.599179] Lustre: Skipped 1 previous similar message [ 1947.953125] Lustre: server umount lustre-MDT0000 complete [ 1951.206994] LustreError: 76631:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1951.235869] LustreError: 76631:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1951.674865] LustreError: 76249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788801916 with bad export cookie 9045482694168623633 [ 1951.686225] LustreError: 76249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 1951.686547] LustreError: MGC192.168.206.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1952.088207] Lustre: server umount lustre-MDT0001 complete [ 1962.256916] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing unload_modules_local [ 1964.760724] Key type lgssc unregistered [ 1965.028521] LNet: 78166:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1965.035888] LNetError: 78166:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1965.062618] LNet: Removed LNI 192.168.206.156@tcp [ 1965.832767] Key type .llcrypt unregistered [ 1965.835087] Key type ._llcrypt unregistered [ 1985.525267] Key type ._llcrypt registered [ 1985.529834] Key type .llcrypt registered [ 1985.630470] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing load_modules_local [ 1986.435456] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1986.566082] alg: No test for adler32 (adler32-zlib) [ 1987.695514] Lustre: Lustre: Build Version: 2.17.58_2_gefa5002 [ 1987.973130] LNet: Added LNI 192.168.206.156@tcp [8/256/0/180] [ 1989.679187] Key type lgssc registered [ 1990.709703] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1997.325651] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1997.341679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1998.741775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2002.604297] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 2009.057658] LustreError: 79553:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2009.087336] LustreError: 79553:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2010.062761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2010.686906] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2015.006391] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 2021.437740] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2026.921570] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2033.214055] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2033.655538] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2039.642508] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 2046.250866] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2051.295623] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2052.041843] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2053.578996] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2053.588259] Lustre: Skipped 1 previous similar message [ 2056.186498] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2056.197394] Lustre: Skipped 2 previous similar messages [ 2060.421713] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2060.426386] Lustre: Skipped 4 previous similar messages [ 2068.459207] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2068.465571] Lustre: Skipped 9 previous similar messages [ 2085.105899] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2085.110911] Lustre: Skipped 22 previous similar messages [ 2117.229359] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2117.235926] Lustre: Skipped 34 previous similar messages [ 2141.937971] Lustre: 87648:0:(mgs_llog.c:1462:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2144.206747] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 13:28:27 (1788802107) [ 2145.468925] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2146.863935] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 13:28:30 (1788802110) [ 2151.392016] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2151.398673] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2151.404931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2151.410171] Lustre: Skipped 1 previous similar message [ 2156.295238] Lustre: server umount lustre-OST0000 complete [ 2161.644463] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2161.646196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2161.665721] Lustre: Skipped 1 previous similar message [ 2161.674106] Lustre: Skipped 1 previous similar message [ 2166.308636] Lustre: server umount lustre-MDT0000 complete [ 2166.762251] LustreError: 79554:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2166.786225] LustreError: 79554:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2169.560793] LustreError: 87650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788802134 with bad export cookie 14401636234623916375 [ 2169.561948] LustreError: MGC192.168.206.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2169.568632] LustreError: 87650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2169.866861] Lustre: server umount lustre-MDT0001 complete [ 2178.594862] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing unload_modules_local [ 2180.822254] Key type lgssc unregistered [ 2181.097131] LNet: 88897:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2181.102146] LNetError: 88897:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2181.115052] LNet: Removed LNI 192.168.206.156@tcp [ 2181.925596] Key type .llcrypt unregistered [ 2181.929783] Key type ._llcrypt unregistered [ 2185.794469] Key type ._llcrypt registered [ 2185.797682] Key type .llcrypt registered [ 2185.865831] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2189.301490] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing load_modules [ 2190.179087] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2190.213255] alg: No test for adler32 (adler32-zlib) [ 2191.269386] Lustre: Lustre: Build Version: 2.17.58_2_gefa5002 [ 2191.469365] LNet: Added LNI 192.168.206.156@tcp [8/256/0/180] [ 2193.103241] Key type lgssc registered [ 2194.066654] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2195.179396] LustreError: 90094:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2209.619796] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing load_modules_local [ 2217.184851] LustreError: 90094:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2217.669685] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2217.682608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2219.058950] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2222.629241] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 2229.225086] LustreError: 91120:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2229.234855] LustreError: 91120:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2229.440847] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 13:29:52 (1788802192) [ 2233.339612] LustreError: 91151:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2233.349857] LustreError: 91151:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2233.355809] LustreError: 91151:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 15, retries 0, failed: rc = -5 [ 2233.590426] Lustre: server umount lustre-MDT0000 complete [ 2245.047899] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing unload_modules_local [ 2247.319971] Key type lgssc unregistered [ 2247.589756] LNet: 92643:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2247.601906] LNetError: 92643:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2247.629775] LNet: Removed LNI 192.168.206.156@tcp [ 2248.492627] Key type .llcrypt unregistered [ 2248.495300] Key type ._llcrypt unregistered [ 2264.018411] Key type ._llcrypt registered [ 2264.021305] Key type .llcrypt registered [ 2264.173932] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing load_modules_local [ 2264.929768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2264.962437] alg: No test for adler32 (adler32-zlib) [ 2266.006782] Lustre: Lustre: Build Version: 2.17.58_2_gefa5002 [ 2266.194780] LNet: Added LNI 192.168.206.156@tcp [8/256/0/180] [ 2267.847789] Key type lgssc registered [ 2268.574655] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2273.932821] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2273.947770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2275.297358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2278.966969] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 2284.517530] LustreError: 93857:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2284.549401] LustreError: 93857:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2285.901947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2286.470602] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2290.063596] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 2295.972670] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2300.614813] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2306.333507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2306.759001] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2311.466978] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 2317.397971] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2363.609580] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 13:32:06 (1788802326) [ 2365.116212] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2366.936723] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 13:32:10 (1788802330) [ 2371.553273] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2371.555152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2371.566995] Lustre: Skipped 1 previous similar message [ 2371.585219] Lustre: Skipped 2 previous similar messages [ 2376.677384] LustreError: 95297:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2376.687508] LustreError: 95297:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2376.739886] Lustre: server umount lustre-MDT0000 complete [ 2379.787430] LustreError: 93837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788802344 with bad export cookie 11106870917350801751 [ 2379.790810] LustreError: MGC192.168.206.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2379.796274] LustreError: 93837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2380.170961] Lustre: server umount lustre-MDT0001 complete [ 2394.258746] Lustre: server umount lustre-OST0000 complete [ 2405.965101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2406.458682] LustreError: 98131:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2406.706489] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2412.001136] LustreError: 98132:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2412.517522] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 2417.123287] LustreError: 98131:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2421.430582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2422.047475] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2426.864954] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 2433.114745] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2437.974384] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2449.429917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2449.636608] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2449.641901] LustreError: 98121:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2449.661268] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2449.669841] LustreError: 100214:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -98 [ 2449.677390] LustreError: 100214:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2449.686857] LustreError: 100214:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2449.767491] Lustre: server umount lustre-OST0000 complete [ 2449.771644] LustreError: 100214:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2456.929851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2457.108463] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2457.121755] LustreError: 98119:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2457.169374] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2457.176240] LustreError: 100679:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -98 [ 2457.185512] LustreError: 100679:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2457.194226] LustreError: 100679:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2457.251768] Lustre: server umount lustre-OST0000 complete [ 2457.254475] LustreError: 100679:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2466.854765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2467.035582] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2467.042203] Lustre: Found index 0 for lustre-OST0000, updating log [ 2467.055101] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2467.123571] Lustre: lustre-OST0000: new disk, initializing [ 2467.126267] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2467.131628] Lustre: 101214:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2467.313871] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2473.959711] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 2478.717419] Lustre: Failing over lustre-OST0000 [ 2479.256384] Lustre: server umount lustre-OST0000 complete [ 2486.756739] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2486.759791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2486.776084] Lustre: Skipped 2 previous similar messages [ 2486.790969] Lustre: Skipped 1 previous similar message [ 2491.873373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2491.886653] Lustre: Skipped 1 previous similar message [ 2494.035975] Lustre: server umount lustre-MDT0000 complete [ 2496.992643] LustreError: 98131:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2497.027812] LustreError: 98131:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2498.015631] LustreError: 98112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788802462 with bad export cookie 11106870917350804495 [ 2498.015741] Lustre: 93169:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788802446/real 1788802446] req@00000000bef4d1cd x1875695087218560/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788802462 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2498.029198] LustreError: MGC192.168.206.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2498.042090] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2498.475324] Lustre: server umount lustre-MDT0001 complete [ 2506.670427] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 13:34:29 (1788802469) [ 2523.496792] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing load_modules_local [ 2533.192658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2533.453662] LustreError: 103560:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2533.555903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2537.839333] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 2545.301546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2550.318653] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 2553.141706] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2560.268025] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2560.941659] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2567.016504] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 2571.187789] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2576.090841] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2576.192158] LustreError: lustre-OST0001: New target registration disabled. Use 'lctl set_param allow_register=unlimited' to allow new target registrations. [ 2576.214390] LustreError: lustre-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 2576.222525] LustreError: 105865:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -13 [ 2576.233442] LustreError: 105865:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0001 [ 2576.239063] LustreError: 105865:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0001 not registered [ 2576.266524] Lustre: server umount lustre-OST0001 complete [ 2576.270048] LustreError: 105865:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 2578.440273] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2578.441354] Lustre: lustre-OST0000: Recovery over after 0:07, of 2 clients 2 recovered and 0 were evicted. [ 2578.481436] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2584.548565] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2595.321315] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2599.534594] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.56@tcp (stopping) [ 2602.464413] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2602.473878] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2604.017620] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2604.033977] Lustre: Skipped 1 previous similar message [ 2605.627325] Lustre: server umount lustre-MDT0001 complete [ 2607.010443] LustreError: 104604:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2607.043710] LustreError: 104604:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2615.806450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2615.889548] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2615.893381] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2615.932278] Lustre: 103550:0:(mgs_llog.c:1462:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2616.641853] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2616.649062] Lustre: Skipped 2 previous similar messages [ 2617.720423] LustreError: 107069:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 2617.725923] LustreError: 107069:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 2617.732792] LustreError: 103552:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2617.802062] LustreError: 107076:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.156@tcp: cfg command failed: rc = -17 [ 2617.808671] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2617.808671] [ 2620.476457] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 2621.945516] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2621.953364] Lustre: Skipped 1 previous similar message [ 2630.276319] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 13:36:33 (1788802593) [ 2646.607857] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing load_modules_local [ 2662.650264] Lustre: 109105:0:(mgs_llog.c:1462:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2678.638320] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2678.740861] LustreError: lustre-OST0001: New target registration disabled. Use 'lctl set_param allow_register=unlimited' to allow new target registrations. [ 2678.759708] LustreError: lustre-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 2678.769263] LustreError: 109897:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -13 [ 2678.779208] LustreError: 109897:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0001 [ 2678.788054] LustreError: 109897:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0001 not registered [ 2678.810859] Lustre: server umount lustre-OST0001 complete [ 2678.816919] LustreError: 109897:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 2686.455808] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2690.150157] Lustre: 110242:0:(mgs_llog.c:1462:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2785.248916] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2785.257366] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2785.276851] Lustre: Skipped 1 previous similar message [ 2785.286197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2785.295538] Lustre: Skipped 1 previous similar message [ 2791.116910] Lustre: server umount lustre-MDT0000 complete [ 2794.793555] LustreError: 103542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788802759 with bad export cookie 11106870917350805503 [ 2794.797306] LustreError: MGC192.168.206.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2794.810298] LustreError: 103542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 2795.157773] Lustre: server umount lustre-MDT0001 complete [ 2808.972352] Lustre: server umount lustre-OST0000 complete [ 2816.668580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2825.926639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2833.647398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2843.447456] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2863.547484] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing load_modules_local [ 2872.352585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2872.378696] Lustre: lustre-MDT0000: reset Object Index mappings [ 2872.774205] LustreError: 114156:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2872.794984] LustreError: 114156:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2872.864448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2877.411837] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 2887.714469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2887.731243] Lustre: lustre-MDT0001: reset Object Index mappings [ 2893.517453] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 2897.054727] Lustre: 115301:0:(mgs_llog.c:1462:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2904.210613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2911.739014] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 2912.492886] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:398 to 0x280000400:417) [ 2917.636303] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:535 to 0x280000401:578) [ 2920.055728] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2920.113821] LustreError: lustre-OST0001: New target registration disabled. Use 'lctl set_param allow_register=unlimited' to allow new target registrations. [ 2920.123608] LustreError: lustre-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 2920.129973] LustreError: 116469:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -13 [ 2920.135908] LustreError: 116469:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0001 [ 2920.140254] LustreError: 116469:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0001 not registered [ 2920.158406] Lustre: server umount lustre-OST0001 complete [ 2920.162847] LustreError: 116469:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 2927.994661] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2931.624505] Lustre: 116816:0:(mgs_llog.c:1462:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2979.948922] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 13:42:23 (1788802943) [ 3005.920993] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3005.929600] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3005.940052] Lustre: Skipped 2 previous similar messages [ 3005.948671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3005.954093] Lustre: Skipped 5 previous similar messages [ 3011.414435] Lustre: server umount lustre-MDT0000 complete [ 3014.795438] LustreError: 114138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788802979 with bad export cookie 11106870917350852130 [ 3014.801980] LustreError: MGC192.168.206.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3014.808626] LustreError: 114138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3015.137916] LustreError: 114152:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3015.149075] LustreError: 114152:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3021.392987] Lustre: server umount lustre-MDT0001 complete [ 3031.438819] Lustre: server umount lustre-OST0000 complete [ 3040.177990] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_hostid [ 3049.498424] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing load_modules_local [ 3083.056633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3083.350290] Lustre: First target registration detected: opening a one-time 24-hour registration window. Use 'lctl set_param allow_register=0' to close it early, or 'lctl set_param -P allow_register=0' to lock down permanently. [ 3083.395042] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3083.438806] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3083.573994] Lustre: lustre-MDT0000: new disk, initializing [ 3083.729977] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3083.745259] Lustre: Skipped 2 previous similar messages [ 3083.774461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3088.742650] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 3096.038936] Lustre: server umount lustre-MDT0000 complete [ 3105.907430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3106.158573] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3106.180652] Lustre: 121034:0:(mgs_llog.c:1462:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3106.198569] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3110.812646] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 3115.868737] Lustre: server umount lustre-MDT0000 complete [ 3124.742728] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 13:44:48 (1788803088) [ 3126.299809] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3127.991735] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 13:44:51 (1788803091) [ 3143.858479] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing load_modules_local [ 3153.319392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3157.616234] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 3166.792222] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3167.017358] Lustre: 123084:0:(mgs_llog.c:1462:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3167.151354] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3167.156234] Lustre: Skipped 1 previous similar message [ 3167.293316] Lustre: lustre-MDT0001: new disk, initializing [ 3167.400994] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3167.412530] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3172.045813] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 3176.828310] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3182.784981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3183.022445] Lustre: lustre-OST0000: new disk, initializing [ 3183.028413] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3183.033666] Lustre: 124663:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3184.825424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3184.836325] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3184.919335] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3187.844285] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 3197.403579] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3197.562202] Lustre: lustre-OST0001: new disk, initializing [ 3197.565314] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3197.574301] Lustre: 125536:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3199.240138] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3199.253266] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3199.365501] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3203.243747] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 3211.440645] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3215.131161] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3581.291508] Lustre: 131945:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3581.297771] Lustre: 131945:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3581.307086] Lustre: 131945:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3581.319773] Lustre: 131945:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3581.327416] Lustre: 131945:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3581.333158] Lustre: 131945:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3879.598838] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 13:57:22 (1788803842) [ 3894.752845] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3894.771984] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3894.791149] Lustre: Skipped 3 previous similar messages [ 3894.804636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3894.814146] Lustre: Skipped 5 previous similar messages [ 3900.894879] Lustre: server umount lustre-MDT0000 complete [ 3900.899401] LustreError: 124656:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3900.921260] LustreError: 124656:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3904.261362] LustreError: 123078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788803868 with bad export cookie 11106870917350918742 [ 3904.272233] LustreError: MGC192.168.206.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3904.276818] LustreError: 123078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3906.020660] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3906.033904] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3906.037283] Lustre: Skipped 4 previous similar messages [ 3906.040582] Lustre: Skipped 4 previous similar messages [ 3910.666821] Lustre: server umount lustre-MDT0001 complete [ 3920.650210] Lustre: server umount lustre-OST0000 complete [ 3932.826881] Lustre: server umount lustre-OST0001 complete [ 3941.414050] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_hostid [ 3951.143897] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing load_modules_local [ 3986.712139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3986.996873] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3987.046366] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3987.175325] Lustre: lustre-MDT0000: new disk, initializing [ 3987.344880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3987.352678] Lustre: Skipped 5 previous similar messages [ 3987.372529] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3992.269120] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 4003.273034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4003.374913] Lustre: 135001:0:(mgs_llog.c:1462:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4003.423586] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4003.427510] Lustre: Skipped 1 previous similar message [ 4003.484923] Lustre: lustre-MDT0001: new disk, initializing [ 4003.581815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4003.596938] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4007.951405] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 4017.525978] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4023.765918] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4030.630514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4030.967791] Lustre: lustre-OST0000: new disk, initializing [ 4030.971388] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4030.980908] Lustre: 136966:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4031.066239] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4031.072753] Lustre: Skipped 1 previous similar message [ 4032.654291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4032.681510] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4032.846730] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4037.963470] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 4048.163797] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4058.033369] Lustre: lustre-MDD0000: changelog on [ 4060.937584] Lustre: lustre-MDD0001: changelog on [ 5857.834921] Lustre: lustre-MDD0001: changelog off [ 5858.987608] Lustre: 135008:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 5859.565411] Lustre: lustre-MDD0000: changelog off [ 5865.444132] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5865.451640] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5865.462066] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5865.465172] Lustre: Skipped 3 previous similar messages [ 5865.955361] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5869.411211] Lustre: server umount lustre-OST0000 complete [ 5872.609672] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5872.618330] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5872.629523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5872.633431] Lustre: Skipped 1 previous similar message [ 5876.192378] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5876.193757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5876.211313] Lustre: Skipped 1 previous similar message [ 5878.372671] Lustre: server umount lustre-MDT0000 complete [ 5880.771691] LustreError: 134992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788805845 with bad export cookie 11106870917351740003 [ 5880.773969] LustreError: MGC192.168.206.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5880.780719] LustreError: 134992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5881.004215] Lustre: server umount lustre-MDT0001 complete [ 5888.021311] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing unload_modules_local [ 5889.808682] Key type lgssc unregistered [ 5890.009336] LNet: 146104:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5890.012793] LNetError: 146104:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5890.020924] LNet: Removed LNI 192.168.206.156@tcp [ 5890.444244] Key type .llcrypt unregistered [ 5890.447062] Key type ._llcrypt unregistered [ 5901.514305] Key type ._llcrypt registered [ 5901.516507] Key type .llcrypt registered [ 5901.589558] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_hostid [ 5910.448717] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing load_modules_local [ 5910.951430] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5910.974594] alg: No test for adler32 (adler32-zlib) [ 5911.907450] Lustre: Lustre: Build Version: 2.17.58_2_gefa5002 [ 5912.060720] LNet: Added LNI 192.168.206.156@tcp [8/256/0/180] [ 5913.695178] Key type lgssc registered [ 5914.414311] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5933.384454] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5933.395873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5934.526364] Lustre: First target registration detected: opening a one-time 24-hour registration window. Use 'lctl set_param allow_register=0' to close it early, or 'lctl set_param -P allow_register=0' to lock down permanently. [ 5934.547633] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5934.574239] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5934.635941] Lustre: lustre-MDT0000: new disk, initializing [ 5934.691144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5934.705142] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5937.215123] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 5943.896685] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5943.993605] Lustre: 149278:0:(mgs_llog.c:1462:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5944.039439] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5944.043296] Lustre: Skipped 1 previous similar message [ 5944.092265] Lustre: lustre-MDT0001: new disk, initializing [ 5944.156436] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5944.185411] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5944.191925] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5946.526848] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 5951.594087] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5954.436784] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5958.440811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5958.625633] Lustre: lustre-OST0000: new disk, initializing [ 5958.630361] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5958.634379] Lustre: 151244:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5958.685515] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5961.786101] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 5966.822671] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5968.903346] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5968.909143] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5968.972623] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5976.661500] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing load_module obdecho/obdecho [ 5977.628414] LustreError: 152247:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 5977.634077] LustreError: 152247:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 5978.570447] LustreError: 152343:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 5984.225757] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5984.229434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5984.232640] Lustre: Skipped 2 previous similar messages [ 5984.235412] Lustre: Skipped 1 previous similar message [ 5989.379550] Lustre: server umount lustre-MDT0000 complete [ 5991.220681] LustreError: 149268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788805955 with bad export cookie 11281019474667995268 [ 5991.223350] LustreError: MGC192.168.206.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5991.229860] LustreError: 149268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5991.399741] Lustre: server umount lustre-MDT0001 complete [ 6003.785113] Lustre: server umount lustre-OST0000 complete [ 6016.095319] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_hostid [ 6021.956617] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing load_modules_local [ 6041.696397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6041.810537] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6041.826666] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6041.867300] Lustre: lustre-MDT0000: new disk, initializing [ 6041.909074] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6041.918871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6043.982501] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 6050.049727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6050.126316] Lustre: 156053:0:(mgs_llog.c:1462:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6050.161495] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6050.164151] Lustre: Skipped 1 previous similar message [ 6050.207750] Lustre: lustre-MDT0001: new disk, initializing [ 6050.267771] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6050.289630] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6050.294354] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6052.684504] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 6058.404968] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6061.517102] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6065.686671] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6065.857838] Lustre: lustre-OST0000: new disk, initializing [ 6065.860801] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6065.865854] Lustre: 158020:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6065.933658] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6069.618094] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 6074.800315] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6075.920467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6075.927593] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6075.994846] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6077.219528] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6077.360877] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6079.655176] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6079.767635] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6080.993546] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6080.997270] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6081.001855] LustreError: Skipped 1 previous similar message [ 6081.002858] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6081.008926] Lustre: Skipped 1 previous similar message [ 6081.017510] Lustre: Skipped 1 previous similar message [ 6082.855957] Lustre: server umount lustre-OST0000 complete [ 6086.115100] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6086.115284] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6086.119246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6086.129805] Lustre: Skipped 1 previous similar message [ 6091.003044] Lustre: server umount lustre-MDT0000 complete [ 6091.233049] LustreError: 157616:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6091.244372] LustreError: 157616:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6092.682551] LustreError: 156847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788806057 with bad export cookie 11281019474667996227 [ 6092.684745] LustreError: MGC192.168.206.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6092.688626] LustreError: 156847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6092.821873] Lustre: server umount lustre-MDT0001 complete [ 6094.386654] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 14:34:18 (1788806058) [ 6095.045392] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6095.868967] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 14:34:19 (1788806059) [ 6099.644607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6099.878380] LustreError: 160428:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6099.916158] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6102.577615] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 6105.057767] LustreError: 160429:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6107.693573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6110.051898] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 6113.683430] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6116.415340] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6119.678083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6122.922207] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 6126.273679] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6142.035203] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.56@tcp (stopping) [ 6142.038125] Lustre: Skipped 1 previous similar message [ 6143.456477] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6143.462923] Lustre: Skipped 2 previous similar messages [ 6145.972842] Lustre: server umount lustre-MDT0000 complete [ 6147.146053] LustreError: 162918:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6147.904352] LustreError: 160408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788806112 with bad export cookie 11281019474667997095 [ 6147.908886] LustreError: MGC192.168.206.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6147.913451] LustreError: 160408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6148.576323] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6148.584755] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6148.587602] Lustre: Skipped 3 previous similar messages [ 6152.266494] LustreError: 160424:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6152.274477] LustreError: 160424:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6157.385678] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.56@tcp (stopping) [ 6157.389531] Lustre: Skipped 2 previous similar messages [ 6162.400049] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6162.497730] Lustre: server umount lustre-MDT0001 complete [ 6165.308982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6171.232862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6171.370489] LustreError: 164291:0:(llog.c:1655:llog_backup()) MGC192.168.206.156@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6171.375533] Lustre: 164291:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.156@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6182.919547] LustreError: 164296:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6182.928074] LustreError: 164296:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6199.372556] LustreError: 164296:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6199.387283] LustreError: 164296:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 6232.543254] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6232.546356] Lustre: 164291:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.156@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6233.568179] LustreError: 164314:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6233.579721] LustreError: 164314:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 23 previous similar messages [ 6273.515425] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6273.519510] Lustre: Skipped 2 previous similar messages [ 6275.635321] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 6278.633372] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6279.813440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6292.964098] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x9c8e3b715bd9a0f2 [ 6292.970533] Lustre: MGC192.168.206.156@tcp: Connection restored to 0@lo (at 0@lo) [ 6293.055786] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -11 [ 6295.331666] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 6298.601351] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6298.718082] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6299.514862] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6301.954305] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6302.755867] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6313.952127] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6313.955301] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6313.962340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6313.969964] Lustre: Skipped 1 previous similar message [ 6313.974807] Lustre: Skipped 1 previous similar message [ 6319.605194] Lustre: server umount lustre-MDT0000 complete [ 6321.534717] LustreError: 160408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788806286 with bad export cookie 11281019474668003570 [ 6321.535432] LustreError: MGC192.168.206.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6321.540662] LustreError: 160408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6321.715113] Lustre: server umount lustre-MDT0001 complete [ 6334.083666] Lustre: server umount lustre-OST0000 complete [ 6344.038865] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_hostid [ 6349.372744] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing load_modules_local [ 6366.257828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6366.379342] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6366.401214] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6366.455785] Lustre: lustre-MDT0000: new disk, initializing [ 6366.500111] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6366.503417] Lustre: Skipped 1 previous similar message [ 6366.511316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6368.703937] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 6374.516237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6374.576967] Lustre: 169370:0:(mgs_llog.c:1462:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6374.739192] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6376.635985] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 6381.734230] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6384.372294] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6387.884787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6388.035828] Lustre: 171333:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6391.155252] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 6391.816059] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6391.862574] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6395.945225] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6398.153210] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6398.278259] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6400.450995] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6400.552371] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6402.017916] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6402.019781] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6402.021661] LustreError: Skipped 1 previous similar message [ 6402.028018] Lustre: Skipped 1 previous similar message [ 6402.028680] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6402.035709] Lustre: Skipped 6 previous similar messages [ 6407.594502] Lustre: server umount lustre-OST0000 complete [ 6410.721194] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6415.560380] Lustre: server umount lustre-MDT0000 complete [ 6417.384500] LustreError: 169379:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6417.394360] LustreError: 169379:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 44 previous similar messages [ 6417.537510] LustreError: 170161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788806382 with bad export cookie 11281019474668020370 [ 6417.539509] LustreError: MGC192.168.206.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6417.549252] LustreError: 170161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6417.732592] Lustre: server umount lustre-MDT0001 complete [ 6421.939634] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 14:39:45 (1788806385) [ 6425.179983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6427.753977] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 6432.445753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6434.717411] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 6438.483535] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6441.348944] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6444.771076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6448.019111] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 6451.558202] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6465.697426] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 14:40:29 (1788806429) [ 6467.555659] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6467.556821] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6467.570508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6467.575564] Lustre: Skipped 4 previous similar messages [ 6467.581634] Lustre: Skipped 5 previous similar messages [ 6473.434590] Lustre: server umount lustre-OST0000 complete [ 6481.469213] Lustre: server umount lustre-MDT0000 complete [ 6483.278439] LustreError: 173723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788806447 with bad export cookie 11281019474668021252 [ 6483.282120] LustreError: MGC192.168.206.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6483.283382] LustreError: 173723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6483.458708] Lustre: server umount lustre-MDT0001 complete [ 6488.503592] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing unload_modules_local [ 6490.044846] Key type lgssc unregistered [ 6490.209600] LNet: 177484:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6490.217017] LNetError: 177484:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6490.227850] LNet: Removed LNI 192.168.206.156@tcp [ 6490.673600] Key type .llcrypt unregistered [ 6490.675072] Key type ._llcrypt unregistered [ 6499.619802] Key type ._llcrypt registered [ 6499.621596] Key type .llcrypt registered [ 6499.676615] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing load_modules_local [ 6500.096671] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6500.137138] alg: No test for adler32 (adler32-zlib) [ 6501.017610] Lustre: Lustre: Build Version: 2.17.58_2_gefa5002 [ 6501.123850] LNet: Added LNI 192.168.206.156@tcp [8/256/0/180] [ 6502.719156] Key type lgssc registered [ 6503.207618] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6506.504232] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6506.514792] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6522.015633] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6527.135744] LustreError: 178700:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.156@tcp: failed processing log, type 4: rc = -110 [ 6532.255676] LustreError: 178700:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6532.266768] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 6532.269831] LustreError: MGC192.168.206.156@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 6532.278249] LustreError: 178700:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 6532.282536] LustreError: 178700:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -22 [ 6532.285423] LustreError: 178700:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 6532.315684] Lustre: server umount lustre-OST0000 complete [ 6532.317523] LustreError: 178700:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6536.537323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6537.805749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6540.025370] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 6544.304646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6544.531595] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6546.520624] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 6549.921566] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6552.485624] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6555.707885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6555.809752] LustreError: 181011:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6555.815216] Lustre: 181011:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.206.156@tcp: invalid local config log lustre-OST0000: rc = -22 [ 6555.887430] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6558.443264] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 6561.815265] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6564.832048] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6564.836835] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6564.846084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6566.368833] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6566.369176] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6566.371556] Lustre: Skipped 1 previous similar message [ 6566.384698] Lustre: Skipped 1 previous similar message [ 6569.443448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6569.447245] Lustre: Skipped 2 previous similar messages [ 6574.561138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6574.564096] Lustre: Skipped 1 previous similar message [ 6577.119114] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6577.198845] Lustre: server umount lustre-MDT0000 complete [ 6578.697845] LustreError: 181038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788806543 with bad export cookie 12435420981484178860 [ 6578.700065] LustreError: MGC192.168.206.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6578.702289] LustreError: 181038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6579.679522] LustreError: 179929:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000009a2fa364 x1875699527866112/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6579.680166] LustreError: 179196:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6579.680436] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6579.680983] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6579.708116] LustreError: 179196:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6579.758645] Lustre: server umount lustre-MDT0001 complete [ 6587.481428] Lustre: server umount lustre-OST0000 complete [ 6591.275484] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_hostid [ 6595.422305] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing load_modules_local [ 6610.326978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6610.423299] Lustre: First target registration detected: opening a one-time 24-hour registration window. Use 'lctl set_param allow_register=0' to close it early, or 'lctl set_param -P allow_register=0' to lock down permanently. [ 6610.435310] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6610.451477] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6610.491601] Lustre: lustre-MDT0000: new disk, initializing [ 6610.525527] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6610.533414] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6612.193596] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 6617.095747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6617.141974] Lustre: 184530:0:(mgs_llog.c:1462:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6617.168599] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6617.171025] Lustre: Skipped 1 previous similar message [ 6617.209866] Lustre: lustre-MDT0001: new disk, initializing [ 6617.249299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6617.268115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6617.272137] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6618.918177] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 6623.032540] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6625.468994] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6628.444417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6628.573507] Lustre: lustre-OST0000: new disk, initializing [ 6628.576073] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6628.580561] Lustre: 186500:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6628.630204] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6630.993996] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 6635.427128] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6637.059829] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6637.064972] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6637.110968] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6637.265378] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6637.358575] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6639.053706] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6639.170922] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6642.147555] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6642.147745] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6642.156347] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6642.158038] LustreError: Skipped 1 previous similar message [ 6642.163493] Lustre: Skipped 1 previous similar message [ 6646.111678] Lustre: server umount lustre-OST0000 complete [ 6648.287856] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6648.290664] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6648.296497] Lustre: Skipped 1 previous similar message [ 6653.670285] Lustre: server umount lustre-MDT0000 complete [ 6655.066436] LustreError: 186499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788806619 with bad export cookie 12435420981484179553 [ 6655.070793] LustreError: MGC192.168.206.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6655.071299] LustreError: 186499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6655.201085] Lustre: server umount lustre-MDT0001 complete [ 6658.563287] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 14:43:42 (1788806622) [ 6665.350533] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing unload_modules_local [ 6666.628522] Key type lgssc unregistered [ 6666.765606] LNet: 189441:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6666.768821] LNetError: 189441:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6666.778313] LNet: Removed LNI 192.168.206.156@tcp [ 6667.162805] Key type .llcrypt unregistered [ 6667.164195] Key type ._llcrypt unregistered [ 6675.454128] Key type ._llcrypt registered [ 6675.455096] Key type .llcrypt registered [ 6675.513485] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing load_modules_local [ 6675.916976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6675.947114] alg: No test for adler32 (adler32-zlib) [ 6676.843844] Lustre: Lustre: Build Version: 2.17.58_2_gefa5002 [ 6676.989521] LNet: Added LNI 192.168.206.156@tcp [8/256/0/180] [ 6678.600561] Key type lgssc registered [ 6679.048677] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6682.254399] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6682.263648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6683.441908] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6685.065155] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 6688.699248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6688.849948] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6690.492695] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 6693.365543] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6695.615480] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6698.748529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6698.857144] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6698.858766] LustreError: 192491:0:(llog.c:1696:llog_backup()) MGC192.168.206.156@tcp: failed to backup log lustre-OST0000: rc = -28 [ 6698.861911] Lustre: 192491:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.206.156@tcp: can't backup local config lustre-OST0000: rc = -28 [ 6698.917643] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6701.490668] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 6704.532167] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6709.215975] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6709.218732] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6709.224499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6710.241631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6710.243309] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6710.248148] Lustre: Skipped 1 previous similar message [ 6711.507345] Lustre: server umount lustre-MDT0000 complete [ 6712.943128] LustreError: 190611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788806677 with bad export cookie 3435077255985169184 [ 6712.947530] LustreError: MGC192.168.206.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6712.947851] LustreError: 190611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6713.070344] Lustre: server umount lustre-MDT0001 complete [ 6724.766262] Lustre: server umount lustre-OST0000 complete [ 6728.490751] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_hostid [ 6732.635262] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing load_modules_local [ 6746.882614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6746.972734] Lustre: First target registration detected: opening a one-time 24-hour registration window. Use 'lctl set_param allow_register=0' to close it early, or 'lctl set_param -P allow_register=0' to lock down permanently. [ 6746.983870] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6746.995835] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6747.038522] Lustre: lustre-MDT0000: new disk, initializing [ 6747.076173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6747.084101] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6748.849549] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 6753.337314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6753.391086] Lustre: 196011:0:(mgs_llog.c:1462:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6753.417569] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6753.421513] Lustre: Skipped 1 previous similar message [ 6753.423633] LustreError: 196791:0:(llog.c:1696:llog_backup()) MGC192.168.206.156@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 6753.427616] LustreError: 196791:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 6753.431066] Lustre: 196791:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.156@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 6753.442197] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6753.445115] Lustre: Skipped 1 previous similar message [ 6753.487839] Lustre: lustre-MDT0001: new disk, initializing [ 6753.533485] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6753.550894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6753.556226] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6755.092872] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 6759.261669] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6761.477420] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6764.131091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6764.241343] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6764.242775] Lustre: Skipped 1 previous similar message [ 6764.244374] LustreError: 197950:0:(llog.c:1696:llog_backup()) MGC192.168.206.156@tcp: failed to backup log lustre-OST0000: rc = -28 [ 6764.249098] LustreError: 197950:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 6764.251937] Lustre: 197950:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.156@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 6764.255897] Lustre: 197950:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 6764.266452] Lustre: lustre-OST0000: new disk, initializing [ 6764.269254] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6764.273175] Lustre: 197975:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6764.305794] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6766.433309] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 6767.816531] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6767.820724] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6767.872158] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6770.650863] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6772.337403] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6772.432580] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6774.211958] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6774.307862] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6778.338657] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6778.341485] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6778.342059] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6778.342065] Lustre: Skipped 2 previous similar messages [ 6778.343896] Lustre: Skipped 1 previous similar message [ 6778.353483] LustreError: Skipped 1 previous similar message [ 6781.203603] Lustre: server umount lustre-OST0000 complete [ 6783.457226] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6783.458164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6783.463235] Lustre: Skipped 1 previous similar message [ 6783.465588] Lustre: Skipped 1 previous similar message [ 6788.576238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6788.579068] Lustre: Skipped 1 previous similar message [ 6797.279190] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6797.346527] Lustre: server umount lustre-MDT0000 complete [ 6798.781629] LustreError: 197974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788806763 with bad export cookie 3435077255985169870 [ 6798.784584] LustreError: MGC192.168.206.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6798.788829] LustreError: 197974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6798.938450] Lustre: server umount lustre-MDT0001 complete [ 6802.352852] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 14:46:06 (1788806766) [ 6808.672679] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing load_modules_local [ 6812.376451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6812.534162] LustreError: 201100:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6812.541049] LustreError: 201100:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6812.548612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6814.145214] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 6817.473672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6819.169934] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 6820.222946] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6822.652529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6822.752714] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 6822.757095] Lustre: Skipped 2 previous similar messages [ 6824.895774] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 6828.253346] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6828.326363] Lustre: lustre-OST0001: new disk, initializing [ 6828.328610] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6828.333070] Lustre: 203404:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6830.711805] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 6830.806651] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6830.812689] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6830.849229] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6836.613332] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6843.479394] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6844.163364] Lustre: DEBUG MARKER: ADD OST3 [ 6846.276658] loop: module loaded [ 6846.296675] loop0: detected capacity change from 0 to 204800000 [ 6846.379148] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6846.384016] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6846.392153] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6848.910738] loop0: detected capacity change from 0 to 204800000 [ 6851.128807] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6851.234240] Lustre: lustre-OST0003: new disk, initializing [ 6851.236444] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 6851.241805] Lustre: 205134:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6851.305452] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 6851.308358] Lustre: Skipped 3 previous similar messages [ 6851.576533] Lustre: DEBUG MARKER: STOP OST3 [ 6852.394925] Lustre: Failing over lustre-OST0003 [ 6852.432140] Lustre: server umount lustre-OST0003 complete [ 6852.840703] LustreError: 202585:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.206.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6853.807763] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 6857.508519] Lustre: DEBUG MARKER: START OST3 again [ 6858.317222] LustreError: 202584:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.206.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6858.324311] LustreError: 202584:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6860.255259] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6862.807442] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 6865.390691] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 6865.394531] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 6865.411167] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000401 [ 6876.935955] Lustre: Failing over lustre-OST0003 [ 6876.979667] Lustre: server umount lustre-OST0003 complete [ 6880.746307] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6880.748254] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6880.750134] LustreError: Skipped 1 previous similar message [ 6880.757564] LustreError: 202584:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6880.764980] LustreError: 202584:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6885.862592] LustreError: 204511:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6885.870151] LustreError: 204511:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6890.977307] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6890.980650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6890.982120] Lustre: Skipped 4 previous similar messages [ 6890.984236] Lustre: Skipped 2 previous similar messages [ 6895.971938] Lustre: server umount lustre-MDT0000 complete [ 6896.097735] LustreError: 201101:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6896.105651] LustreError: 201101:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6897.519979] LustreError: 201083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788806862 with bad export cookie 3435077255985170752 [ 6897.521132] LustreError: MGC192.168.206.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6897.526679] LustreError: 201083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6903.750995] Lustre: server umount lustre-MDT0001 complete [ 6905.310283] Lustre: server umount lustre-OST0000 complete [ 6906.781785] Lustre: server umount lustre-OST0001 complete [ 6909.309841] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_hostid [ 6912.445727] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing load_modules_local [ 6926.794714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6926.906850] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6926.921493] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6926.958840] Lustre: lustre-MDT0000: new disk, initializing [ 6926.999990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6927.003068] Lustre: Skipped 1 previous similar message [ 6927.013192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6928.775989] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 6933.290355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6933.336710] Lustre: 210544:0:(mgs_llog.c:1462:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6933.448857] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6934.851406] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 6939.081213] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6941.325278] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6944.055848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6944.178538] Lustre: lustre-OST0000: new disk, initializing [ 6944.180305] Lustre: Skipped 1 previous similar message [ 6944.182646] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6944.184835] Lustre: Skipped 2 previous similar messages [ 6944.187422] Lustre: 212513:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6945.924384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6945.928725] Lustre: Skipped 1 previous similar message [ 6945.931701] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6945.975738] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6946.569186] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 6950.830890] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6952.457888] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6952.542794] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6954.109564] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6954.184176] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6956.515961] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6956.517729] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6956.520218] LustreError: Skipped 1 previous similar message [ 6956.526118] Lustre: Skipped 2 previous similar messages [ 6956.527754] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6956.534949] Lustre: Skipped 5 previous similar messages [ 6969.311106] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6969.363529] Lustre: server umount lustre-OST0000 complete [ 6976.726129] Lustre: server umount lustre-MDT0000 complete [ 6976.991831] LustreError: 210556:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6976.996617] LustreError: 210556:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6977.945359] LustreError: 210536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788806942 with bad export cookie 3435077255985172985 [ 6977.950865] LustreError: MGC192.168.206.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6978.061158] Lustre: server umount lustre-MDT0001 complete [ 6979.322121] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 14:49:03 (1788806943) [ 6985.014814] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_hostid [ 6988.757735] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing load_modules_local [ 7001.757410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7001.865742] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7001.878668] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7001.917935] Lustre: lustre-MDT0000: new disk, initializing [ 7001.946875] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7003.473855] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 7007.962449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7008.007482] Lustre: 217114:0:(mgs_llog.c:1462:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7008.126821] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7009.671129] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 7013.669397] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7015.657439] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7018.274238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7018.396584] Lustre: 219082:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7020.335398] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7020.794401] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 7024.907598] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7026.637202] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7026.736176] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7028.390405] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7028.477791] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7030.756667] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7030.756698] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7030.757525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7030.757529] Lustre: Skipped 6 previous similar messages [ 7030.759747] LustreError: Skipped 1 previous similar message [ 7030.769164] Lustre: Skipped 4 previous similar messages [ 7035.426718] Lustre: server umount lustre-OST0000 complete [ 7038.944340] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7043.317786] Lustre: server umount lustre-MDT0000 complete [ 7044.931671] LustreError: 217108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788807009 with bad export cookie 3435077255985173867 [ 7044.932411] LustreError: MGC192.168.206.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7044.941089] LustreError: 217108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 7045.113464] Lustre: server umount lustre-MDT0001 complete [ 7048.937879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7049.150403] LustreError: 221300:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7051.038783] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 7054.695447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7056.472738] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 7059.290440] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7061.424752] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7063.966993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7064.102641] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7064.105628] Lustre: Skipped 7 previous similar messages [ 7066.167446] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 7068.742453] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7083.490638] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7086.804533] Lustre: server umount lustre-OST0000 complete [ 7094.425390] Lustre: server umount lustre-MDT0000 complete [ 7095.860763] LustreError: 221282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788807060 with bad export cookie 3435077255985174749 [ 7095.861842] LustreError: MGC192.168.206.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7095.866223] LustreError: 221282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7095.988354] Lustre: server umount lustre-MDT0001 complete [ 7202.638533] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing unload_modules_local [ 7203.892412] Key type lgssc unregistered [ 7204.035462] LNet: 224982:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7204.038386] LNetError: 224982:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7204.045521] LNet: Removed LNI 192.168.206.156@tcp [ 7204.454169] Key type .llcrypt unregistered [ 7204.455600] Key type ._llcrypt unregistered [ 7213.975135] Key type ._llcrypt registered [ 7213.976683] Key type .llcrypt registered [ 7214.016319] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_hostid [ 7220.612448] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing load_modules_local [ 7221.075235] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7221.087862] alg: No test for adler32 (adler32-zlib) [ 7221.957606] Lustre: Lustre: Build Version: 2.17.58_2_gefa5002 [ 7222.048037] LNet: Added LNI 192.168.206.156@tcp [8/256/0/180] [ 7223.631188] Key type lgssc registered [ 7224.081428] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7237.261909] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7237.268026] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7238.348701] Lustre: First target registration detected: opening a one-time 24-hour registration window. Use 'lctl set_param allow_register=0' to close it early, or 'lctl set_param -P allow_register=0' to lock down permanently. [ 7238.358520] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7238.373258] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7238.410896] Lustre: lustre-MDT0000: new disk, initializing [ 7238.433340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7238.441807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7239.897156] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 7244.084389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7244.132343] Lustre: 228303:0:(mgs_llog.c:1462:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7244.158857] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7244.161419] Lustre: Skipped 1 previous similar message [ 7244.196110] Lustre: lustre-MDT0001: new disk, initializing [ 7244.227105] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7244.242698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7244.246326] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7245.652238] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 7249.567090] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7251.570612] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7253.894651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7254.011314] Lustre: lustre-OST0000: new disk, initializing [ 7254.013399] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7254.016487] Lustre: 230271:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7254.052126] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7256.154479] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 7257.209872] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7257.213693] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7257.248123] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7260.063754] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7261.504476] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7261.581607] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7263.027582] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7263.106390] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7267.810274] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7267.810525] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7267.811676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7267.813629] LustreError: Skipped 1 previous similar message [ 7267.820237] Lustre: Skipped 1 previous similar message [ 7272.929585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7272.931987] Lustre: Skipped 2 previous similar messages [ 7278.047120] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7278.051026] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7278.107681] Lustre: server umount lustre-OST0000 complete [ 7280.096200] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7280.100425] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7280.105701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7280.108312] Lustre: Skipped 1 previous similar message [ 7283.170171] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7285.670857] Lustre: server umount lustre-MDT0000 complete [ 7287.161644] LustreError: 228293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788807251 with bad export cookie 188630954431659281 [ 7287.165798] LustreError: MGC192.168.206.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7287.166302] LustreError: 228293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7287.305761] Lustre: server umount lustre-MDT0001 complete [ 7294.048382] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing load_modules_local [ 7298.331809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7298.490443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7299.983797] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 7303.386796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7303.522315] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7304.969536] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 7305.997432] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7308.197964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7310.174537] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 7312.959936] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7313.025089] Lustre: lustre-OST0001: new disk, initializing [ 7313.026843] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7313.029085] Lustre: 235506:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7313.062922] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7313.065324] Lustre: Skipped 1 previous similar message [ 7314.969097] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 7315.393569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7315.396727] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7315.431868] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7320.606305] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7326.401995] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7330.503182] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 14:54:54 (1788807294) [ 7334.367911] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7334.370774] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7334.375683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7334.377881] Lustre: Skipped 2 previous similar messages [ 7337.368192] Lustre: server umount lustre-MDT0000 complete [ 7338.603137] LustreError: 235505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788807303 with bad export cookie 188630954431660163 [ 7338.606847] LustreError: MGC192.168.206.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7338.607404] LustreError: 235505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7338.722477] Lustre: server umount lustre-MDT0001 complete [ 7350.211063] Lustre: server umount lustre-OST0000 complete [ 7361.690721] Lustre: server umount lustre-OST0001 complete [ 7363.787357] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_hostid [ 7367.112954] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing load_modules_local [ 7379.697994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7379.783416] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7379.795594] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7379.835184] Lustre: lustre-MDT0000: new disk, initializing [ 7379.859876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7379.866689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7381.366365] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 7385.814064] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7385.853303] Lustre: 239445:0:(mgs_llog.c:1462:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7385.953726] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7387.342734] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 7391.102530] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7393.017159] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7395.369625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7395.480337] Lustre: lustre-OST0000: new disk, initializing [ 7395.481926] Lustre: Skipped 1 previous similar message [ 7395.483731] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7395.486618] Lustre: Skipped 2 previous similar messages [ 7395.489381] Lustre: 241413:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7397.372531] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7397.376693] Lustre: Skipped 1 previous similar message [ 7397.378803] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7397.409459] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7397.438435] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 7401.327844] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7402.851614] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7402.935063] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7404.364254] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7404.442302] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7407.585675] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7407.586390] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7407.588982] LustreError: Skipped 1 previous similar message [ 7407.592009] Lustre: Skipped 3 previous similar messages [ 7407.592592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7407.597451] Lustre: Skipped 5 previous similar messages [ 7419.359089] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7419.409974] Lustre: server umount lustre-OST0000 complete [ 7421.919794] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7421.922646] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7421.928269] Lustre: Skipped 1 previous similar message [ 7426.785054] Lustre: server umount lustre-MDT0000 complete [ 7428.065533] LustreError: 240255:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7428.073575] LustreError: 240255:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7428.123761] LustreError: 239436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788807392 with bad export cookie 188630954431662256 [ 7428.128075] LustreError: MGC192.168.206.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7428.128990] LustreError: 239436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7428.246613] Lustre: server umount lustre-MDT0001 complete [ 7431.334530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7431.520821] LustreError: 243629:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7431.552372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7431.555650] Lustre: Skipped 2 previous similar messages [ 7432.932132] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 7435.781694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7437.328378] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 7439.796336] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7441.827282] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7444.000955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7446.077199] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 7448.486713] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7594.976792] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7594.980646] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7594.985921] Lustre: Skipped 1 previous similar message [ 7594.987896] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7594.990537] Lustre: Skipped 7 previous similar messages [ 7599.440670] Lustre: server umount lustre-OST0000 complete [ 7606.802746] Lustre: server umount lustre-MDT0000 complete [ 7607.949102] LustreError: 243611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788807572 with bad export cookie 188630954431663138 [ 7607.951917] LustreError: 243611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7607.952352] LustreError: MGC192.168.206.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7608.070301] Lustre: server umount lustre-MDT0001 complete [ 7723.492548] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing unload_modules_local [ 7724.529820] Key type lgssc unregistered [ 7724.660468] LNet: 247245:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7724.663270] LNetError: 247245:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7724.676445] LNet: Removed LNI 192.168.206.156@tcp [ 7725.012128] Key type .llcrypt unregistered [ 7725.013609] Key type ._llcrypt unregistered [ 7733.676941] Key type ._llcrypt registered [ 7733.678458] Key type .llcrypt registered [ 7733.719802] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_hostid [ 7739.791908] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing load_modules_local [ 7740.097232] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7740.117267] alg: No test for adler32 (adler32-zlib) [ 7740.971254] Lustre: Lustre: Build Version: 2.17.58_2_gefa5002 [ 7741.066567] LNet: Added LNI 192.168.206.156@tcp [8/256/0/180] [ 7742.647189] Key type lgssc registered [ 7742.991610] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7756.746781] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing load_modules_local [ 7760.300930] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7760.308876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7761.394520] Lustre: First target registration detected: opening a one-time 24-hour registration window. Use 'lctl set_param allow_register=0' to close it early, or 'lctl set_param -P allow_register=0' to lock down permanently. [ 7761.404614] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7761.417097] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7761.450145] Lustre: lustre-MDT0000: new disk, initializing [ 7761.470087] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7761.476712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7762.891738] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 7767.223630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7767.275712] Lustre: 251280:0:(mgs_llog.c:1462:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7767.303437] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7767.305884] Lustre: Skipped 1 previous similar message [ 7767.346265] Lustre: lustre-MDT0001: new disk, initializing [ 7767.393595] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7767.410485] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7767.417081] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7768.879319] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 7771.315838] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7773.729641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7773.840086] Lustre: lustre-OST0000: new disk, initializing [ 7773.842319] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7773.846325] Lustre: 252911:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7773.879539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7776.087473] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 7779.835413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7779.840161] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7779.880489] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7780.455967] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7780.523782] Lustre: lustre-OST0001: new disk, initializing [ 7780.526675] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7780.530664] Lustre: 253786:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7780.566712] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7782.537300] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 7787.990689] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7788.549448] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7788.553226] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7788.593420] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7794.663633] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7798.239821] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7798.241793] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7798.245306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7798.754844] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7798.759236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7798.761240] Lustre: Skipped 2 previous similar messages [ 7803.616416] Lustre: server umount lustre-MDT0000 complete [ 7804.828933] LustreError: 251273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788807769 with bad export cookie 1728504269487931926 [ 7804.832885] LustreError: MGC192.168.206.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7804.833315] LustreError: 251273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7804.940940] Lustre: server umount lustre-MDT0001 complete [ 7816.197561] Lustre: server umount lustre-OST0000 complete [ 7827.737339] Lustre: server umount lustre-OST0001 complete [ 7832.014788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7834.890216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7837.499724] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7840.578101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7846.543530] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing load_modules_local [ 7849.881175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7849.888234] Lustre: lustre-MDT0000: reset Object Index mappings [ 7850.018484] LustreError: 258781:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7850.024650] LustreError: 258781:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 7850.031768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7851.358021] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 7854.137870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7854.144675] Lustre: lustre-MDT0001: reset Object Index mappings [ 7855.582614] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 7856.491614] Lustre: 259920:0:(mgs_llog.c:1462:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7858.581788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7858.701951] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7858.704109] Lustre: Skipped 1 previous similar message [ 7859.746763] LustreError: 260275:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7859.751018] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7860.492534] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 7863.245158] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7864.362099] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 7865.185821] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 7868.292603] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7869.444955] Lustre: 261790:0:(mgs_llog.c:1462:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7874.529361] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7874.533027] Lustre: Skipped 5 previous similar messages [ 7874.533437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7874.536253] Lustre: Skipped 3 previous similar messages [ 7876.637147] Lustre: server umount lustre-MDT0000 complete [ 7877.859107] LustreError: 261792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788807842 with bad export cookie 1728504269487938702 [ 7877.862214] LustreError: MGC192.168.206.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7877.862718] LustreError: 261792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7877.971884] Lustre: server umount lustre-MDT0001 complete [ 7888.835204] Lustre: server umount lustre-OST0000 complete [ 7900.257798] Lustre: server umount lustre-OST0001 complete [ 7906.989245] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 15:04:31 (1788807871) [ 7912.304239] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_hostid [ 7915.488967] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing load_modules_local [ 7927.099292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7927.198930] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7927.210847] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7927.247437] Lustre: lustre-MDT0000: new disk, initializing [ 7927.268990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7927.271996] Lustre: Skipped 1 previous similar message [ 7927.277571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7928.595072] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 7932.628025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7932.669204] Lustre: 266047:0:(mgs_llog.c:1462:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7932.769628] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7934.069682] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 7937.741412] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7939.631114] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7941.820488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7941.933126] Lustre: lustre-OST0000: new disk, initializing [ 7941.934619] Lustre: Skipped 1 previous similar message [ 7941.936518] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7941.938654] Lustre: Skipped 2 previous similar messages [ 7941.940962] Lustre: 268014:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7943.289606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7943.291755] Lustre: Skipped 1 previous similar message [ 7943.293590] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7943.326526] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7943.903037] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 7947.463349] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7948.786581] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7948.851033] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7950.140178] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7950.207475] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7953.888748] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7953.889530] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7953.890689] LustreError: Skipped 1 previous similar message [ 7953.895341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7953.897593] Lustre: Skipped 1 previous similar message [ 7959.009072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7959.010742] Lustre: Skipped 1 previous similar message [ 7965.151178] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7965.203383] Lustre: server umount lustre-OST0000 complete [ 7968.735900] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7968.738773] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7968.742174] Lustre: Skipped 1 previous similar message [ 7968.743862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7968.746208] Lustre: Skipped 2 previous similar messages [ 7972.503458] Lustre: server umount lustre-MDT0000 complete [ 7973.695372] LustreError: 266038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788807938 with bad export cookie 1728504269487940774 [ 7973.698150] LustreError: MGC192.168.206.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7973.699840] LustreError: 266038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7973.816090] Lustre: server umount lustre-MDT0001 complete [ 7979.646180] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing load_modules_local [ 7982.973822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7983.095731] LustreError: 270942:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7983.105965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7983.107725] Lustre: Skipped 2 previous similar messages [ 7984.417783] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 7987.186166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7988.593919] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 7989.485753] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7991.562469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7993.540266] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 7996.168041] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7996.226338] Lustre: lustre-OST0001: new disk, initializing [ 7996.227896] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7996.230029] Lustre: 273248:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7998.136707] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 7998.212065] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7998.215158] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7998.258210] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8002.384659] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8003.565662] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8005.359640] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 8005.364588] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 8008.674754] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8008.678126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8008.678472] Lustre: Skipped 3 previous similar messages [ 8008.680714] Lustre: Skipped 3 previous similar messages [ 8014.173351] Lustre: server umount lustre-MDT0000 complete [ 8015.334268] LustreError: 270923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788807980 with bad export cookie 1728504269487941656 [ 8015.337184] LustreError: 270923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8015.337794] LustreError: MGC192.168.206.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8015.444969] Lustre: server umount lustre-MDT0001 complete [ 8027.139926] Lustre: server umount lustre-OST0000 complete [ 8038.619068] Lustre: server umount lustre-OST0001 complete [ 8043.629348] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing load_modules_local [ 8046.819926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8046.935089] LustreError: 276584:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8048.201323] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 8050.907341] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8051.021898] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8051.023810] Lustre: Skipped 4 previous similar messages [ 8052.281157] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 8053.198228] Lustre: 277726:0:(mgs_llog.c:1462:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8055.262392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8056.421196] LustreError: 278081:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8056.429193] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 8057.273281] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 8059.997358] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8061.105087] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 8061.929238] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 8064.873599] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8065.974630] Lustre: 279601:0:(mgs_llog.c:1462:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8067.637572] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 8067.639907] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 8071.574481] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 15:07:15 (1788808035) [ 8087.011456] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8087.012960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8087.014864] Lustre: Skipped 4 previous similar messages [ 8087.016878] Lustre: Skipped 6 previous similar messages [ 8087.019408] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8089.180209] Lustre: server umount lustre-MDT0000 complete [ 8090.410292] LustreError: 277329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788808055 with bad export cookie 1728504269487948026 [ 8090.416462] LustreError: MGC192.168.206.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8092.127925] LustreError: 280490:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8092.134374] LustreError: 280490:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8096.556526] Lustre: server umount lustre-MDT0001 complete [ 8112.095095] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8112.128852] Lustre: server umount lustre-OST0000 complete [ 8119.386447] Lustre: server umount lustre-OST0001 complete [ 8121.344435] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_hostid [ 8124.509831] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing load_modules_local [ 8138.882443] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing load_modules_local [ 8142.209598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8142.281401] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8142.292055] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8142.323264] Lustre: lustre-MDT0000: new disk, initializing [ 8142.345476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8143.593342] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 8147.592570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8147.637094] Lustre: 284122:0:(mgs_llog.c:1462:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8147.737114] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8149.037336] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 8153.505166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8153.602194] Lustre: 285754:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8155.033788] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8155.420889] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 8159.317024] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8161.243937] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 8161.315635] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8165.551487] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8170.304535] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 15:08:54 (1788808134) [ 8171.491565] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8171.493730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8171.495415] Lustre: Skipped 5 previous similar messages [ 8171.497275] Lustre: Skipped 6 previous similar messages [ 8177.052767] Lustre: server umount lustre-MDT0000 complete [ 8178.163171] LustreError: 284916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788808142 with bad export cookie 1728504269487955957 [ 8178.165720] LustreError: 284916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 8178.165924] LustreError: MGC192.168.206.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8178.280438] Lustre: server umount lustre-MDT0001 complete [ 8189.958266] Lustre: server umount lustre-OST0000 complete [ 8200.418334] Lustre: server umount lustre-OST0001 complete [ 8203.345917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8203.519226] LustreError: 288863:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8203.524534] LustreError: 288863:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8203.553471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8203.557111] Lustre: Skipped 6 previous similar messages [ 8204.968289] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 8207.904716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8209.502570] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 8212.024569] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8214.008632] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8216.212855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8218.378981] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 8220.900754] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8233.368666] Lustre: New target registration enabled indefinitely [ 8234.003417] Lustre: New target registration disabled [ 8236.617241] LustreError: 290691:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8236.627160] LustreError: 290691:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 8237.735957] loop1: detected capacity change from 0 to 204800000 [ 8237.841263] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8237.845716] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8237.851224] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8239.671968] loop1: detected capacity change from 0 to 204800000 [ 8241.708982] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8241.747993] LustreError: 288852:0:(mgs_llog.c:685:mgs_find_or_make_fsdb_nolock()) fs27506: New filesystem registration disabled. Use 'lctl set_param allow_register=unlimited' to allow new filesystem registrations: rc = -13 [ 8241.755735] LustreError: 288852:0:(mgs_llog.c:745:mgs_check_index()) Can't get db for fs27506 [ 8241.759275] LustreError: 288852:0:(mgs_handler.c:517:mgs_target_reg()) fs27506-MDT0001: Error checking target index: rc = -13 [ 8241.764494] LustreError: fs27506-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 8241.768641] LustreError: 292599:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -13 [ 8241.772846] LustreError: 292599:0:(tgt_mount.c:2132:server_put_super()) no obd fs27506-MDT0001 [ 8241.776252] LustreError: 292599:0:(tgt_mount.c:116:server_deregister_mount()) fs27506-MDT0001 not registered [ 8241.791390] Lustre: server umount fs27506-MDT0001 complete [ 8241.792905] LustreError: 292599:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 8242.459374] Lustre: New target registration enabled indefinitely [ 8244.695577] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8244.725705] Lustre: Setting parameter fs27506-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs27506-MDT0001 [ 8244.729040] Lustre: Skipped 2 previous similar messages [ 8244.745115] Lustre: srv-fs27506-MDT0001: No data found on store. Initialize space. [ 8244.747869] Lustre: Skipped 4 previous similar messages [ 8244.785904] Lustre: fs27506-MDT0001: new disk, initializing [ 8244.787212] Lustre: Skipped 3 previous similar messages [ 8244.826094] Lustre: srv-fs27506-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 8244.829600] Lustre: 293065:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs27506-MDT0001-mdtlov: can't create new llog: rc = -115 [ 8244.832913] LustreError: 293065:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs27506-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 8246.247486] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 8249.173185] Lustre: server umount fs27506-MDT0001 complete [ 8251.916496] Lustre: New target registration disabled [ 8253.512447] loop1: detected capacity change from 0 to 204800000 [ 8255.480231] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8255.606260] Lustre: 294630:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs27506-MDT0001-mdtlov: can't create new llog: rc = -115 [ 8255.608958] LustreError: 294630:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs27506-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 8257.021333] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 8258.702831] Lustre: server umount fs27506-MDT0001 complete [ 8264.671878] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8264.674415] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8268.132799] Lustre: server umount lustre-MDT0000 complete [ 8271.009550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8271.051212] LustreError: MGC192.168.206.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8272.579551] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 8275.273605] loop1: detected capacity change from 0 to 204800000 [ 8276.455224] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8277.169791] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8277.289146] Lustre: 297072:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs27506-MDT0001-mdtlov: can't create new llog: rc = -115 [ 8277.291872] LustreError: 297072:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs27506-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 8278.607756] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 8280.216591] Lustre: server umount fs27506-MDT0001 complete [ 8286.691154] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8292.752669] Lustre: server umount lustre-OST0000 complete [ 8302.048144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8302.050580] Lustre: Skipped 17 previous similar messages [ 8308.191089] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8308.246930] Lustre: server umount lustre-MDT0000 complete [ 8309.408183] LustreError: 288844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788808274 with bad export cookie 1728504269487958631 [ 8309.411029] LustreError: 288844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8309.411386] LustreError: MGC192.168.206.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8309.516164] Lustre: server umount lustre-MDT0001 complete [ 8312.843550] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing unload_modules_local [ 8313.813217] Key type lgssc unregistered [ 8313.926453] LNet: 299363:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8313.928450] LNetError: 299363:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8313.939390] LNet: Removed LNI 192.168.206.156@tcp [ 8314.255122] Key type .llcrypt unregistered [ 8314.255907] Key type ._llcrypt unregistered [ 8321.516577] Key type ._llcrypt registered [ 8321.517811] Key type .llcrypt registered [ 8321.553644] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing load_modules_local [ 8321.849871] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8321.863742] alg: No test for adler32 (adler32-zlib) [ 8322.709890] Lustre: Lustre: Build Version: 2.17.58_2_gefa5002 [ 8322.789919] LNet: Added LNI 192.168.206.156@tcp [8/256/0/180] [ 8324.367129] Key type lgssc registered [ 8324.695323] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8327.059739] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8327.065559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8328.199961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8329.464603] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 8332.135126] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8332.255535] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8333.470344] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 8335.692585] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8337.456793] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8339.592374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8339.704779] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8341.450270] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 8341.732069] LustreError: 302431:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8341.736438] LustreError: 302431:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8343.546295] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8347.108159] LustreError: 302432:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8352.226526] LustreError: 302429:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8358.369741] LustreError: 302851:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8358.376112] LustreError: 302851:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 8359.949696] loop1: detected capacity change from 0 to 204800000 [ 8360.010724] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8360.014723] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8360.018378] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8361.742365] loop1: detected capacity change from 0 to 204800000 [ 8363.494701] LustreError: 302851:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8363.500426] LustreError: 302851:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8363.696188] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8363.733651] Lustre: First target registration detected: opening a one-time 24-hour registration window. Use 'lctl set_param allow_register=0' to close it early, or 'lctl set_param -P allow_register=0' to lock down permanently. [ 8363.762463] Lustre: fs16276-OST0000: new disk, initializing [ 8363.764443] Lustre: srv-fs16276-OST0000: No data found on store. Initialize space. [ 8363.767141] Lustre: 304359:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8363.789886] Lustre: fs16276-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8365.815124] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 8368.678112] Lustre: New target registration disabled [ 8369.295732] Lustre: server umount fs16276-OST0000 complete [ 8373.359879] loop1: detected capacity change from 0 to 204800000 [ 8373.730581] LustreError: 302432:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8373.736595] LustreError: 302432:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 8375.236872] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8375.298985] Lustre: fs16276-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8377.288820] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 8380.166374] loop2: detected capacity change from 0 to 204800000 [ 8380.243765] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8380.247060] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8380.250522] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8382.060796] loop2: detected capacity change from 0 to 204800000 [ 8384.002562] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8384.037798] LustreError: fs16276-OST0001: New target registration disabled. Use 'lctl set_param allow_register=unlimited' to allow new target registrations. [ 8384.042151] LustreError: fs16276-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 8384.047741] LustreError: 307396:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -13 [ 8384.051704] LustreError: 307396:0:(tgt_mount.c:2132:server_put_super()) no obd fs16276-OST0001 [ 8384.055508] LustreError: 307396:0:(tgt_mount.c:116:server_deregister_mount()) fs16276-OST0001 not registered [ 8384.069125] Lustre: server umount fs16276-OST0001 complete [ 8384.071619] LustreError: 307396:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 8385.400177] Lustre: New target registration enabled indefinitely [ 8387.791702] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8387.847198] Lustre: fs16276-OST0001: new disk, initializing [ 8387.848852] Lustre: srv-fs16276-OST0001: No data found on store. Initialize space. [ 8387.852432] Lustre: 307946:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8387.880254] Lustre: fs16276-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8389.915507] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 8392.795271] Lustre: New target registration disabled [ 8393.446680] Lustre: server umount fs16276-OST0000 complete [ 8394.211575] LustreError: 302851:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8394.216657] LustreError: 302851:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 8396.496180] Lustre: server umount fs16276-OST0001 complete [ 8400.581204] loop1: detected capacity change from 0 to 204800000 [ 8402.512159] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8404.523142] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 8406.826853] loop2: detected capacity change from 0 to 204800000 [ 8408.732891] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8408.806335] Lustre: fs16276-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8408.809671] Lustre: Skipped 1 previous similar message [ 8410.770796] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 8413.727436] Lustre: server umount fs16276-OST0000 complete [ 8416.731153] Lustre: server umount fs16276-OST0001 complete [ 8423.903846] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8423.906534] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8423.910758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8424.929265] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8424.929585] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8424.930400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8430.050554] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8430.050583] LustreError: 304987:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8430.052118] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [ 8430.052613] Lustre: Skipped 2 previous similar messages [ 8430.062451] LustreError: 304987:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 18 previous similar messages [ 8434.143130] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8434.200591] Lustre: server umount lustre-OST0000 complete [ 8439.775827] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8439.778767] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8439.782718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8441.556809] Lustre: server umount lustre-MDT0000 complete [ 8442.768318] LustreError: 300587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788808407 with bad export cookie 12708069663022007629 [ 8442.771092] LustreError: MGC192.168.206.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8442.773845] LustreError: 300587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8442.889194] Lustre: server umount lustre-MDT0001 complete [ 8446.654399] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing unload_modules_local [ 8447.680660] Key type lgssc unregistered [ 8447.811293] LNet: 313809:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8447.813341] LNetError: 313809:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8447.821287] LNet: Removed LNI 192.168.206.156@tcp [ 8448.153093] Key type .llcrypt unregistered [ 8448.154245] Key type ._llcrypt unregistered [ 8455.982356] Key type ._llcrypt registered [ 8455.983372] Key type .llcrypt registered [ 8456.025693] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing load_modules_local [ 8456.530509] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8456.540887] alg: No test for adler32 (adler32-zlib) [ 8457.396292] Lustre: Lustre: Build Version: 2.17.58_2_gefa5002 [ 8457.487456] LNet: Added LNI 192.168.206.156@tcp [8/256/0/180] [ 8459.071170] Key type lgssc registered [ 8459.432367] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8461.999258] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8462.005992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8463.149756] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8464.476950] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 8467.317717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8467.465157] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8468.861341] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 8471.388936] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8473.299882] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8475.565700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8475.695068] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8477.734606] LustreError: 317241:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8477.740514] LustreError: 317241:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8477.759447] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 8480.185331] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8482.788497] LustreError: 317241:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8487.908059] LustreError: 316881:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8490.577582] LustreError: 316883:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8490.583142] LustreError: 316883:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8495.289171] Lustre: New target registration enabled indefinitely [ 8495.691172] LustreError: 316882:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8495.696850] LustreError: 316882:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8497.262163] loop1: detected capacity change from 0 to 204800000 [ 8497.324671] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8497.327924] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8497.331606] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8499.106375] loop1: detected capacity change from 0 to 204800000 [ 8501.021499] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8501.074651] Lustre: fs27043-OST0000: new disk, initializing [ 8501.077376] Lustre: srv-fs27043-OST0000: No data found on store. Initialize space. [ 8501.079911] Lustre: 318903:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8501.101539] Lustre: fs27043-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8503.020342] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 8504.292781] LustreError: 317241:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8504.298337] LustreError: 317241:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 8505.584574] Lustre: Registration timeout window started: 60 seconds. New targets can be registered during this window. [ 8506.432261] Lustre: DEBUG MARKER: test157c-1788808456-8221 [ 8521.290553] LustreError: 316881:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8521.297982] LustreError: 316881:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 8555.492359] LustreError: 316883:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8555.496367] LustreError: 316883:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 19 previous similar messages [ 8565.727062] Lustre: New MDT/OST registration window disabled. Use 'lctl set_param allow_register=' to re-enable (default unit: hours). [ 8565.736321] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [ 8568.011992] loop2: detected capacity change from 0 to 204800000 [ 8568.135912] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8568.139410] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8568.143585] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8569.862656] loop2: detected capacity change from 0 to 204800000 [ 8571.729396] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8571.759676] LustreError: fs27043-OST0001: New target registration disabled. Use 'lctl set_param allow_register=unlimited' to allow new target registrations. [ 8571.763517] LustreError: fs27043-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 8571.766292] LustreError: 322782:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -13 [ 8571.768429] LustreError: 322782:0:(tgt_mount.c:2132:server_put_super()) no obd fs27043-OST0001 [ 8571.770340] LustreError: 322782:0:(tgt_mount.c:116:server_deregister_mount()) fs27043-OST0001 not registered [ 8571.779760] Lustre: server umount fs27043-OST0001 complete [ 8571.781137] LustreError: 322782:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 8572.991425] Lustre: New target registration enabled indefinitely [ 8575.218397] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8575.261762] Lustre: fs27043-OST0001: new disk, initializing [ 8575.263467] Lustre: srv-fs27043-OST0001: No data found on store. Initialize space. [ 8575.265883] Lustre: 323340:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8575.287677] Lustre: fs27043-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8575.976675] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [ 8577.270213] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 8582.026820] Lustre: server umount fs27043-OST0000 complete [ 8585.094506] Lustre: server umount fs27043-OST0001 complete [ 8590.304981] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8590.308692] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8590.315068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8591.328736] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8591.330605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8591.334256] Lustre: Skipped 1 previous similar message [ 8594.636938] Lustre: server umount lustre-OST0000 complete [ 8596.450999] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8596.452387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8596.458893] Lustre: Skipped 1 previous similar message [ 8601.568104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8601.570102] Lustre: Skipped 2 previous similar messages [ 8606.688550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8606.690542] Lustre: Skipped 1 previous similar message [ 8610.271075] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8610.342629] Lustre: server umount lustre-MDT0000 complete [ 8611.691189] LustreError: 315037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788808576 with bad export cookie 13069717824491044523 [ 8611.694579] LustreError: 315037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8611.695220] LustreError: MGC192.168.206.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8611.827591] Lustre: server umount lustre-MDT0001 complete [ 8615.596185] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing unload_modules_local [ 8616.718555] Key type lgssc unregistered [ 8616.852440] LNet: 326137:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8616.854722] LNetError: 326137:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8616.864384] LNet: Removed LNI 192.168.206.156@tcp [ 8617.228117] Key type .llcrypt unregistered [ 8617.229150] Key type ._llcrypt unregistered [ 8625.558603] Key type ._llcrypt registered [ 8625.559803] Key type .llcrypt registered [ 8625.596166] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing load_modules_local [ 8625.849022] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8625.916287] alg: No test for adler32 (adler32-zlib) [ 8626.781444] Lustre: Lustre: Build Version: 2.17.58_2_gefa5002 [ 8626.870144] LNet: Added LNI 192.168.206.156@tcp [8/256/0/180] [ 8628.455113] Key type lgssc registered [ 8628.809474] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8631.314752] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8631.319882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8632.473116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8633.826336] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 8636.977945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8637.115138] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8638.517786] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 8641.043621] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8643.294458] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8645.769062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8645.900561] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8646.948350] LustreError: 329259:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8646.954430] LustreError: 329259:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8647.811363] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 8650.384636] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8652.263576] LustreError: 329447:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8657.384555] LustreError: 329446:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8661.078049] LustreError: 329447:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8661.089351] LustreError: 329447:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8662.985159] Lustre: Setting parameter general.allow_register=0 in log params [ 8663.933195] LustreError: 330137:0:(mgs_llog.c:6158:mgs_set_param2()) MGS: persistent allow_register only allowed for 0 or unlimited: rc = -22 [ 8663.935991] LustreError: 330137:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -22 [ 8668.127965] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8668.130289] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8668.134454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8668.641697] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8668.642518] LustreError: 329258:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8668.642888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8668.645058] Lustre: Skipped 1 previous similar message [ 8668.653314] LustreError: 329258:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8671.009211] Lustre: server umount lustre-MDT0000 complete [ 8672.213416] LustreError: 327411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788808636 with bad export cookie 13499100435937796912 [ 8672.216370] LustreError: MGC192.168.206.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8672.217628] LustreError: 327411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8672.340196] Lustre: server umount lustre-MDT0001 complete [ 8692.747698] Lustre: server umount lustre-OST0000 complete [ 8697.596383] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing unload_modules_local [ 8698.715327] Key type lgssc unregistered [ 8698.848722] LNet: 331339:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8698.852782] LNetError: 331339:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8698.861537] LNet: Removed LNI 192.168.206.156@tcp [ 8699.223099] Key type .llcrypt unregistered [ 8699.224232] Key type ._llcrypt unregistered [ 8705.774116] Key type ._llcrypt registered [ 8705.775270] Key type .llcrypt registered [ 8705.808504] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing load_modules_local [ 8706.127666] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8706.137571] alg: No test for adler32 (adler32-zlib) [ 8707.013676] Lustre: Lustre: Build Version: 2.17.58_2_gefa5002 [ 8707.118614] LNet: Added LNI 192.168.206.156@tcp [8/256/0/180] [ 8708.711218] Key type lgssc registered [ 8709.063934] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8712.172881] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8712.179522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8713.325563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8714.616681] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 8716.278747] LustreError: 332583:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 8716.281672] LustreError: 332583:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 8716.284186] LustreError: 332583:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 2, retries 0, failed: rc = -5 [ 8716.418793] Lustre: server umount lustre-MDT0000 complete [ 8721.367762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8728.931493] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing unload_modules_local [ 8729.984493] Key type lgssc unregistered [ 8730.117381] LNet: 335034:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8730.120278] LNetError: 335034:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8730.129280] LNet: Removed LNI 192.168.206.156@tcp [ 8730.436112] Key type .llcrypt unregistered [ 8730.437121] Key type ._llcrypt unregistered [ 8737.845207] Key type ._llcrypt registered [ 8737.847503] Key type .llcrypt registered [ 8737.895330] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing load_modules_local [ 8738.339067] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8738.346621] alg: No test for adler32 (adler32-zlib) [ 8739.211172] Lustre: Lustre: Build Version: 2.17.58_2_gefa5002 [ 8739.301983] LNet: Added LNI 192.168.206.156@tcp [8/256/0/180] [ 8740.879125] Key type lgssc registered [ 8741.221103] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8744.976542] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8744.983708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8746.141937] Lustre: New target registration enabled indefinitely [ 8746.151469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8747.520144] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 8751.008939] Lustre: Disabling parameter general.allow_register= in log params [ 8752.535254] LustreError: 336264:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 8752.539219] LustreError: 336264:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 8752.541815] LustreError: 336264:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -5 [ 8752.663968] Lustre: server umount lustre-MDT0000 complete [ 8757.262226] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing unload_modules_local [ 8758.321078] Key type lgssc unregistered [ 8758.451451] LNet: 337904:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8758.453729] LNetError: 337904:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8758.462292] LNet: Removed LNI 192.168.206.156@tcp [ 8758.812107] Key type .llcrypt unregistered [ 8758.813520] Key type ._llcrypt unregistered [ 8766.394291] Key type ._llcrypt registered [ 8766.395295] Key type .llcrypt registered [ 8766.434311] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing load_modules_local [ 8766.765382] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8766.817043] alg: No test for adler32 (adler32-zlib) [ 8767.672507] Lustre: Lustre: Build Version: 2.17.58_2_gefa5002 [ 8767.765366] LNet: Added LNI 192.168.206.156@tcp [8/256/0/180] [ 8769.343196] Key type lgssc registered [ 8769.732508] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8772.239865] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8772.245890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8773.396821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8774.743033] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 8777.619696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8777.759866] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8779.139667] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 8781.529144] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8783.407603] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8785.721956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8785.836568] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8787.746027] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 8787.876495] LustreError: 340975:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8787.883851] LustreError: 340975:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8790.110127] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8793.061273] LustreError: 341362:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8798.177611] LustreError: 341361:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8800.848531] LustreError: 340976:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8800.854189] LustreError: 340976:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8804.322082] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8804.323520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8804.325866] Lustre: Skipped 2 previous similar messages [ 8804.330911] Lustre: Skipped 2 previous similar messages [ 8809.440973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8809.441860] LustreError: 340975:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8809.443019] Lustre: Skipped 2 previous similar messages [ 8809.449194] LustreError: 340975:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8809.811882] Lustre: server umount lustre-MDT0000 complete [ 8811.004107] LustreError: 341720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788808775 with bad export cookie 16365578673127705812 [ 8811.007842] LustreError: MGC192.168.206.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8811.009176] LustreError: 341720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8811.130238] Lustre: server umount lustre-MDT0001 complete [ 8822.682748] Lustre: server umount lustre-OST0000 complete [ 8827.332428] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing unload_modules_local [ 8828.431513] Key type lgssc unregistered [ 8828.567498] LNet: 342911:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8828.570351] LNetError: 342911:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8828.581284] LNet: Removed LNI 192.168.206.156@tcp [ 8828.942153] Key type .llcrypt unregistered [ 8828.944391] Key type ._llcrypt unregistered [ 8835.881979] Key type ._llcrypt registered [ 8835.883052] Key type .llcrypt registered [ 8835.924979] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing load_modules_local [ 8836.236515] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8836.297753] alg: No test for adler32 (adler32-zlib) [ 8837.163659] Lustre: Lustre: Build Version: 2.17.58_2_gefa5002 [ 8837.261342] LNet: Added LNI 192.168.206.156@tcp [8/256/0/180] [ 8838.855107] Key type lgssc registered [ 8839.221406] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8842.343620] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8842.351450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8843.506708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8844.917787] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 8846.188534] Lustre: Registration timeout window started: 60 seconds. New targets can be registered during this window. [ 8847.144789] LustreError: 344143:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 8847.149766] LustreError: 344143:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 8847.154056] LustreError: 344143:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 3, retries 0, failed: rc = -5 [ 8847.310664] Lustre: server umount lustre-MDT0000 complete [ 8853.872355] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing unload_modules_local [ 8854.881754] Key type lgssc unregistered [ 8855.002411] LNet: 345736:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8855.005340] LNetError: 345736:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8855.013263] LNet: Removed LNI 192.168.206.156@tcp [ 8855.313109] Key type .llcrypt unregistered [ 8855.314781] Key type ._llcrypt unregistered [ 8861.683262] Key type ._llcrypt registered [ 8861.684343] Key type .llcrypt registered [ 8861.715661] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing load_modules_local [ 8862.089397] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8862.101312] alg: No test for adler32 (adler32-zlib) [ 8862.949323] Lustre: Lustre: Build Version: 2.17.58_2_gefa5002 [ 8863.031107] LNet: Added LNI 192.168.206.156@tcp [8/256/0/180] [ 8864.607149] Key type lgssc registered [ 8864.942085] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8867.889691] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8867.895478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8869.036877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8870.416213] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 8871.978537] Lustre: Setting parameter general.allow_register=unlimited in log params [ 8872.624066] LustreError: 346969:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 8872.627707] LustreError: 346969:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 8872.631099] LustreError: 346969:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 4, retries 0, failed: rc = -5 [ 8872.767376] Lustre: server umount lustre-MDT0000 complete [ 8879.001763] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing unload_modules_local [ 8880.084664] Key type lgssc unregistered [ 8880.218442] LNet: 348564:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8880.220678] LNetError: 348564:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8880.228424] LNet: Removed LNI 192.168.206.156@tcp [ 8880.559124] Key type .llcrypt unregistered [ 8880.560712] Key type ._llcrypt unregistered [ 8887.357044] Key type ._llcrypt registered [ 8887.358225] Key type .llcrypt registered [ 8887.398242] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing load_modules_local [ 8887.783550] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8887.864686] alg: No test for adler32 (adler32-zlib) [ 8888.724240] Lustre: Lustre: Build Version: 2.17.58_2_gefa5002 [ 8888.820183] LNet: Added LNI 192.168.206.156@tcp [8/256/0/180] [ 8890.399107] Key type lgssc registered [ 8890.739754] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8893.613793] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8893.621495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8894.781416] Lustre: New target registration enabled indefinitely [ 8894.790789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8896.139879] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 8897.426268] Lustre: Disabling parameter general.allow_register= in log params [ 8899.898548] Lustre: 350529:0:(mgs_llog.c:1462:mgs_modify_param()) MGS: modify general/allow_register= (mode = 1) failed: rc = -2 [ 8899.901466] LustreError: 350529:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 8901.431686] LustreError: 349796:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 8901.435402] LustreError: 349796:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 8901.437845] LustreError: 349796:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -5 [ 8901.555592] Lustre: server umount lustre-MDT0000 complete [ 8906.130523] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing unload_modules_local [ 8907.173485] Key type lgssc unregistered [ 8907.311373] LNet: 351481:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8907.314137] LNetError: 351481:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8907.326383] LNet: Removed LNI 192.168.206.156@tcp [ 8907.634122] Key type .llcrypt unregistered [ 8907.635406] Key type ._llcrypt unregistered [ 8918.081754] Key type ._llcrypt registered [ 8918.083379] Key type .llcrypt registered [ 8918.125914] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_hostid [ 8924.317406] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing load_modules_local [ 8924.666382] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8924.687556] alg: No test for adler32 (adler32-zlib) [ 8925.550135] Lustre: Lustre: Build Version: 2.17.58_2_gefa5002 [ 8925.649670] LNet: Added LNI 192.168.206.156@tcp [8/256/0/180] [ 8927.231217] Key type lgssc registered [ 8927.631013] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8939.611911] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8939.619219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8940.708358] Lustre: First target registration detected: opening a one-time 24-hour registration window. Use 'lctl set_param allow_register=0' to close it early, or 'lctl set_param -P allow_register=0' to lock down permanently. [ 8940.719619] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8940.730462] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8940.768587] Lustre: lustre-MDT0000: new disk, initializing [ 8940.789891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8940.795905] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8942.256185] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 8946.303566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8946.346574] Lustre: 355305:0:(mgs_llog.c:1462:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8946.376527] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8946.378294] Lustre: Skipped 1 previous similar message [ 8946.412404] Lustre: lustre-MDT0001: new disk, initializing [ 8946.446592] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8946.458131] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8946.461575] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8947.936107] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 8951.575440] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8953.318420] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8955.542848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8955.664941] Lustre: lustre-OST0000: new disk, initializing [ 8955.666484] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8955.668844] Lustre: 357275:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8955.701704] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8957.712595] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 8961.518853] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8962.052632] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8962.055988] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8962.096968] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8962.233434] Lustre: Mounted lustre-client - version 2.17.58_2_gefa5002 [ 8962.559259] Lustre: Unmounted lustre-client [ 8967.136419] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8967.136687] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8967.138858] LustreError: Skipped 1 previous similar message [ 8967.139338] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8967.144317] Lustre: Skipped 1 previous similar message [ 8971.479649] Lustre: server umount lustre-OST0000 complete [ 8972.768157] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8972.775050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8972.777603] Lustre: Skipped 1 previous similar message [ 8977.376121] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8977.378377] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8977.382345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8983.008193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8983.010363] Lustre: Skipped 2 previous similar messages [ 8987.103135] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8987.173463] Lustre: server umount lustre-MDT0000 complete [ 8988.503379] LustreError: 355300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788808953 with bad export cookie 3097654147747888393 [ 8988.508710] LustreError: MGC192.168.206.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8988.632936] Lustre: server umount lustre-MDT0001 complete [ 8992.362394] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing unload_modules_local [ 8993.501318] Key type lgssc unregistered [ 9018.207520] LNet: 359433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9018.209804] LNetError: 359433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9095.015324] LNet: Removed LNI 192.168.206.156@tcp [ 9095.308965] Key type .llcrypt unregistered [ 9095.310232] Key type ._llcrypt unregistered [ 9101.414894] Key type ._llcrypt registered [ 9101.415883] Key type .llcrypt registered [ 9101.449662] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_hostid [ 9107.565504] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing load_modules_local [ 9108.073738] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9108.090826] alg: No test for adler32 (adler32-zlib) [ 9108.955589] Lustre: Lustre: Build Version: 2.17.58_2_gefa5002 [ 9109.046373] LNet: Added LNI 192.168.206.156@tcp [8/256/0/180] [ 9110.623179] Key type lgssc registered [ 9111.021312] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9120.809658] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 15:24:45 (1788809085) [ 9123.276146] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9123.283763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9124.363258] Lustre: First target registration detected: opening a one-time 24-hour registration window. Use 'lctl set_param allow_register=0' to close it early, or 'lctl set_param -P allow_register=0' to lock down permanently. [ 9124.371718] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9124.381835] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9124.422451] Lustre: lustre-MDT0000: new disk, initializing [ 9124.442799] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9124.451061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9125.802011] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 9129.910040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9129.954106] Lustre: 362626:0:(mgs_llog.c:1462:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9129.983809] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9129.985443] Lustre: Skipped 1 previous similar message [ 9130.018374] Lustre: lustre-MDT0001: new disk, initializing [ 9130.049898] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9130.061046] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9130.064224] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9131.372231] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 9135.481484] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9137.383032] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9139.699197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9139.808447] Lustre: lustre-OST0000: new disk, initializing [ 9139.810081] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9139.812847] Lustre: 364592:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9139.842834] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9141.938522] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 9145.638847] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9148.918834] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9148.923069] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9148.952336] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9159.140192] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9159.140833] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9159.142056] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9159.144354] Lustre: Skipped 1 previous similar message [ 9159.148560] LustreError: Skipped 1 previous similar message [ 9162.579912] Lustre: server umount lustre-OST0000 complete [ 9164.256763] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9164.257565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9164.260026] Lustre: Skipped 1 previous similar message [ 9164.261546] Lustre: Skipped 1 previous similar message [ 9169.376199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9169.379333] Lustre: Skipped 2 previous similar messages [ 9174.496327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9174.498822] Lustre: Skipped 1 previous similar message [ 9178.079109] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9178.143421] Lustre: server umount lustre-MDT0000 complete [ 9179.427697] LustreError: 364591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788809144 with bad export cookie 16941716676172185443 [ 9179.432483] LustreError: MGC192.168.206.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9179.567106] Lustre: server umount lustre-MDT0001 complete [ 9183.437591] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing unload_modules_local [ 9184.547757] Key type lgssc unregistered [ 9184.679312] LNet: 366499:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9184.683100] LNetError: 366499:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9184.690310] LNet: Removed LNI 192.168.206.156@tcp [ 9185.011957] Key type .llcrypt unregistered [ 9185.013716] Key type ._llcrypt unregistered [ 9193.138481] Key type ._llcrypt registered [ 9193.139548] Key type .llcrypt registered [ 9193.173171] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing load_modules_local [ 9193.495952] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9193.542041] alg: No test for adler32 (adler32-zlib) [ 9194.393351] Lustre: Lustre: Build Version: 2.17.58_2_gefa5002 [ 9194.487762] LNet: Added LNI 192.168.206.156@tcp [8/256/0/180] [ 9196.063099] Key type lgssc registered [ 9196.414092] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9198.942148] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9198.947833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9200.093061] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9201.479409] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 9204.405965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9204.542231] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9205.928511] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 9208.295572] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9210.058174] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9212.334171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9212.458572] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9214.387672] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 9216.806426] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9230.561131] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9231.035243] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9232.461367] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.56@tcp (stopping) [ 9234.518730] Lustre: server umount lustre-MDT0000 complete [ 9234.912966] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9234.915104] LustreError: 367743:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9234.917030] Lustre: Skipped 2 previous similar messages [ 9234.922148] LustreError: 367743:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9235.760054] LustreError: 367727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788809200 with bad export cookie 9928004985701569640 [ 9235.763638] LustreError: MGC192.168.206.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9235.763786] LustreError: 367727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9237.575605] LustreError: 367742:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9237.581062] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.56@tcp (stopping) [ 9240.032138] LustreError: 367743:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9240.032406] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9240.043162] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9241.920349] Lustre: server umount lustre-MDT0001 complete [ 9247.822502] Lustre: lustre-OST0000: Not available for connect from 192.168.206.56@tcp (stopping) [ 9249.497564] Lustre: server umount lustre-OST0000 complete [ 9252.401525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9253.857513] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 9254.791694] Lustre: 371941:0:(mgs_llog.c:1820:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.156@tcp [ 9255.467508] Lustre: server umount MGS complete [ 9258.290094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9258.449609] LustreError: 372511:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9258.474034] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9260.052232] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 9262.922323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9263.036639] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9264.439577] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 9267.282589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9269.450216] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 9271.890441] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9272.394154] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 9274.172513] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9274.695478] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9278.943840] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9278.947087] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9278.951897] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9283.043912] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9283.046710] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9284.687194] Lustre: server umount lustre-OST0000 complete [ 9288.161873] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9288.169312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9288.172928] Lustre: Skipped 3 previous similar messages [ 9292.079345] Lustre: server umount lustre-MDT0000 complete [ 9293.280521] LustreError: 372511:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9293.289623] LustreError: 372511:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9293.399390] LustreError: 372536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788809258 with bad export cookie 9928004985701570466 [ 9293.402623] LustreError: MGC192.168.206.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9293.404942] LustreError: 372536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9293.522131] Lustre: server umount lustre-MDT0001 complete [ 9297.204051] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing unload_modules_local [ 9298.383262] Key type lgssc unregistered [ 9298.509506] LNet: 375945:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9298.513959] LNetError: 375945:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9298.522447] LNet: Removed LNI 192.168.206.156@tcp [ 9298.898680] Key type .llcrypt unregistered [ 9298.900354] Key type ._llcrypt unregistered [ 9306.293848] Key type ._llcrypt registered [ 9306.295356] Key type .llcrypt registered [ 9306.344217] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_hostid [ 9313.731468] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing load_modules_local [ 9314.058777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9314.074070] alg: No test for adler32 (adler32-zlib) [ 9314.918962] Lustre: Lustre: Build Version: 2.17.58_2_gefa5002 [ 9315.010113] LNet: Added LNI 192.168.206.156@tcp [8/256/0/180] [ 9316.591165] Key type lgssc registered [ 9316.972103] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9332.479525] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9332.493456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9333.601303] Lustre: First target registration detected: opening a one-time 24-hour registration window. Use 'lctl set_param allow_register=0' to close it early, or 'lctl set_param -P allow_register=0' to lock down permanently. [ 9333.612408] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9333.623930] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9333.664289] Lustre: lustre-MDT0000: new disk, initializing [ 9333.691356] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9333.698162] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9335.510786] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 9340.897208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9340.962579] Lustre: 379074:0:(mgs_llog.c:1462:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9340.997917] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9340.999945] Lustre: Skipped 1 previous similar message [ 9341.039781] Lustre: lustre-MDT0001: new disk, initializing [ 9341.077613] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9341.093212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9341.096875] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9342.903924] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 9347.081206] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9349.025259] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9351.454836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9351.560755] Lustre: lustre-OST0000: new disk, initializing [ 9351.563246] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9351.566092] Lustre: 381042:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9351.593945] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9353.511923] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 9357.495675] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9359.038162] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9359.865122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9359.869976] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9359.903546] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9360.157466] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 9361.847389] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9361.977611] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9364.964040] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9364.965023] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9364.967050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9364.969576] Lustre: Skipped 1 previous similar message [ 9364.975029] Lustre: Skipped 1 previous similar message [ 9368.974234] Lustre: server umount lustre-OST0000 complete [ 9372.128246] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9372.130962] LustreError: Skipped 1 previous similar message [ 9372.132555] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9372.137354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9375.202350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9375.203595] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9376.461983] Lustre: server umount lustre-MDT0000 complete [ 9378.328110] LustreError: 379068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788809343 with bad export cookie 12239039567417704168 [ 9378.336120] LustreError: MGC192.168.206.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9378.497913] Lustre: server umount lustre-MDT0001 complete [ 9382.915981] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 15:29:06 (1788809346) [ 9389.217850] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_hostid [ 9392.699716] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing load_modules_local [ 9405.718836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9405.816699] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9405.828960] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9405.878822] Lustre: lustre-MDT0000: new disk, initializing [ 9405.902170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9405.910900] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9407.359111] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 9411.695894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9411.732832] Lustre: 385798:0:(mgs_llog.c:1462:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9411.752513] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9411.754049] Lustre: Skipped 1 previous similar message [ 9411.789042] Lustre: lustre-MDT0001: new disk, initializing [ 9411.816567] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9411.828130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9411.831268] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9413.421583] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 9417.282481] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9419.256465] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9421.750096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9421.896564] Lustre: lustre-OST0000: new disk, initializing [ 9421.898114] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9421.900323] Lustre: 387764:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9421.942045] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9424.564406] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 9427.450421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9427.453679] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9427.481774] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9428.819056] Lustre: DEBUG MARKER: oleg656-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9442.787915] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9442.787945] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9442.793052] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9442.796867] Lustre: Skipped 1 previous similar message [ 9442.798932] Lustre: Skipped 1 previous similar message [ 9445.846560] Lustre: server umount lustre-OST0000 complete [ 9447.906199] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9447.906444] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9447.907441] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9447.907446] Lustre: Skipped 1 previous similar message [ 9447.908410] LustreError: Skipped 1 previous similar message [ 9447.914600] Lustre: Skipped 1 previous similar message [ 9453.308700] Lustre: server umount lustre-MDT0000 complete [ 9454.937373] LustreError: 386592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788809419 with bad export cookie 12239039567417705022 [ 9454.940844] LustreError: MGC192.168.206.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9454.941018] LustreError: 386592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 9455.066112] Lustre: server umount lustre-MDT0001 complete [ 9459.361270] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing unload_modules_local [ 9460.450143] Key type lgssc unregistered [ 9460.584423] LNet: 389912:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9460.586974] LNetError: 389912:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9460.593446] LNet: Removed LNI 192.168.206.156@tcp [ 9460.954404] Key type .llcrypt unregistered [ 9460.955613] Key type ._llcrypt unregistered [ 9496.253141] Key type ._llcrypt registered [ 9496.254368] Key type .llcrypt registered [ 9496.292422] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing unload_modules_local [ 9496.402331] Key type .llcrypt unregistered [ 9496.403756] Key type ._llcrypt unregistered [ 9524.245472] Key type ._llcrypt registered [ 9524.246426] Key type .llcrypt registered [ 9524.281858] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing unload_modules_local [ 9524.382441] Key type .llcrypt unregistered [ 9524.383737] Key type ._llcrypt unregistered [ 9535.905892] Key type ._llcrypt registered [ 9535.907794] Key type .llcrypt registered [ 9535.945526] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing unload_modules_local [ 9536.044450] Key type .llcrypt unregistered [ 9536.045703] Key type ._llcrypt unregistered [ 9548.571081] Key type ._llcrypt registered [ 9548.572076] Key type .llcrypt registered [ 9548.609190] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing unload_modules_local [ 9548.714215] Key type .llcrypt unregistered [ 9548.716047] Key type ._llcrypt unregistered [ 9578.653743] Key type ._llcrypt registered [ 9578.654952] Key type .llcrypt registered [ 9578.701645] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing unload_modules_local [ 9578.827259] Key type .llcrypt unregistered [ 9578.828623] Key type ._llcrypt unregistered [ 9592.908053] Key type ._llcrypt registered [ 9592.909053] Key type .llcrypt registered [ 9592.949467] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing unload_modules_local [ 9593.071310] Key type .llcrypt unregistered [ 9593.073508] Key type ._llcrypt unregistered [ 9606.985643] Key type ._llcrypt registered [ 9606.986802] Key type .llcrypt registered [ 9607.027832] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing unload_modules_local [ 9607.151598] Key type .llcrypt unregistered [ 9607.153191] Key type ._llcrypt unregistered [ 9619.646016] Key type ._llcrypt registered [ 9619.647465] Key type .llcrypt registered [ 9619.687160] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing unload_modules_local [ 9619.821371] Key type .llcrypt unregistered [ 9619.823142] Key type ._llcrypt unregistered [ 9633.079520] Key type ._llcrypt registered [ 9633.080603] Key type .llcrypt registered [ 9633.115627] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing unload_modules_local [ 9633.230466] Key type .llcrypt unregistered [ 9633.232279] Key type ._llcrypt unregistered [ 9645.800499] Key type ._llcrypt registered [ 9645.801470] Key type .llcrypt registered [ 9645.837211] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing unload_modules_local [ 9645.936209] Key type .llcrypt unregistered [ 9645.937280] Key type ._llcrypt unregistered [ 9671.529206] Key type ._llcrypt registered [ 9671.530641] Key type .llcrypt registered [ 9671.587887] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing unload_modules_local [ 9671.744315] Key type .llcrypt unregistered [ 9671.746945] Key type ._llcrypt unregistered [ 9684.237703] Key type ._llcrypt registered [ 9684.239300] Key type .llcrypt registered [ 9684.302194] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing unload_modules_local [ 9684.449331] Key type .llcrypt unregistered [ 9684.451303] Key type ._llcrypt unregistered [ 9695.552300] Key type ._llcrypt registered [ 9695.553276] Key type .llcrypt registered [ 9695.585269] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing unload_modules_local [ 9695.679350] Key type .llcrypt unregistered [ 9695.681270] Key type ._llcrypt unregistered [ 9716.352202] Key type ._llcrypt registered [ 9716.353218] Key type .llcrypt registered [ 9716.391452] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_hostid [ 9724.033943] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing load_modules_local [ 9724.370281] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9724.409052] alg: No test for adler32 (adler32-zlib) [ 9725.256384] Lustre: Lustre: Build Version: 2.17.58_2_gefa5002 [ 9725.364465] LNet: Added LNI 192.168.206.156@tcp [8/256/0/180] [ 9726.943207] Key type lgssc registered [ 9727.510790] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9741.195877] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9741.202908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9742.290232] Lustre: First target registration detected: opening a one-time 24-hour registration window. Use 'lctl set_param allow_register=0' to close it early, or 'lctl set_param -P allow_register=0' to lock down permanently. [ 9742.299430] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [ 9742.310156] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [ 9742.357753] Lustre: tst250fs-MDT0000: new disk, initializing [ 9742.381525] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9742.388934] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9744.009339] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 9748.953425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9749.015796] Lustre: 409763:0:(mgs_llog.c:1462:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9749.050002] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [ 9749.053724] Lustre: Skipped 1 previous similar message [ 9749.095536] Lustre: tst250fs-MDT0001: new disk, initializing [ 9749.139203] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9749.158513] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9749.164515] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9750.593167] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 9753.017853] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9755.784700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9755.898591] Lustre: tst250fs-OST0000: new disk, initializing [ 9755.900550] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [ 9755.903376] Lustre: 411394:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9755.932633] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9758.214356] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 9762.321873] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9762.330242] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9762.404738] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9762.869841] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9762.988169] Lustre: tst250fs-OST0001: new disk, initializing [ 9762.992806] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [ 9762.999153] Lustre: 412266:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9763.045514] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9765.191895] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_default_debug -1 all [ 9767.744782] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [ 9768.649767] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9768.653294] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9768.695175] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9779.177383] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9779.179767] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 9779.185436] Lustre: Skipped 3 previous similar messages [ 9779.192812] Lustre: Skipped 3 previous similar messages [ 9784.292201] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 9784.297448] Lustre: Skipped 2 previous similar messages [ 9784.658814] Lustre: server umount tst250fs-MDT0000 complete [ 9786.340187] LustreError: 409756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788809751 with bad export cookie 2543280192145910390 [ 9786.346308] LustreError: MGC192.168.206.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9786.348052] LustreError: 409756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9786.507792] Lustre: server umount tst250fs-MDT0001 complete [ 9797.636395] Lustre: server umount tst250fs-OST0000 complete [ 9808.498227] Lustre: server umount tst250fs-OST0001 complete [ 9815.002304] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_hostid [ 9817.897708] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing load_modules_local [ 9829.905608] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 15:36:33 (1788809793) [ 9830.718933] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [ 9837.583762] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing set_hostid [ 9841.462054] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing load_modules_local [ 9852.689287] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 9569 sec ========== 15:36:56 (1788809816) [ 9853.280418] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 15:36:57 (1788809817) === [ 9854.438403] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 15:36:58 (1788809818) === [ 9870.741286] Lustre: DEBUG MARKER: oleg656-server.virtnet: executing unload_modules_local [ 9871.873371] Key type lgssc unregistered [ 9871.996247] LNet: 421398:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9871.999173] LNetError: 421398:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9872.008349] LNet: Removed LNI 192.168.206.156@tcp [ 9872.454128] Key type .llcrypt unregistered [ 9872.455929] Key type ._llcrypt unregistered