[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 496836666 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002354] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004015] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.007023] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.008012] pid_max: default: 32768 minimum: 301 [ 0.009137] LSM: Security Framework initializing [ 0.010060] Yama: becoming mindful. [ 0.011046] SELinux: Initializing. [ 0.012081] *** VALIDATE selinux *** [ 0.020773] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025334] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026168] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027137] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029008] *** VALIDATE tmpfs *** [ 0.031288] *** VALIDATE proc *** [ 0.033035] *** VALIDATE cgroup *** [ 0.034010] *** VALIDATE cgroup2 *** [ 0.035291] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036166] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038039] Spectre V2 : User space: Vulnerable [ 0.039013] Speculative Store Bypass: Vulnerable [ 0.042013] debug: unmapping init [mem 0xffffffff83e59000-0xffffffff83e60fff] [ 0.044970] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045667] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046021] ... version: 2 [ 0.047014] ... bit width: 48 [ 0.048015] ... generic registers: 4 [ 0.049017] ... value mask: 0000ffffffffffff [ 0.050014] ... max period: 00007fffffffffff [ 0.051018] ... fixed-purpose events: 3 [ 0.052022] ... event mask: 000000070000000f [ 0.053333] rcu: Hierarchical SRCU implementation. [ 0.055552] smp: Bringing up secondary CPUs ... [ 0.056647] x86: Booting SMP configuration: [ 0.057030] .... node #0, CPUs: #1 #2 #3 [ 0.061217] smp: Brought up 1 node, 4 CPUs [ 0.062961] smpboot: Max logical packages: 1 [ 0.063019] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.181990] node 0 deferred pages initialised in 116ms [ 0.186485] devtmpfs: initialized [ 0.188418] x86/mm: Memory block size: 128MB [ 0.193000] gcov: version magic: 0x41383552 [ 0.196419] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.200229] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.203510] pinctrl core: initialized pinctrl subsystem [ 0.205278] [ 0.206011] ************************************************************* [ 0.209020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.211018] ** ** [ 0.213027] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.216022] ** ** [ 0.219023] ** This means that this kernel is built to expose internal ** [ 0.221018] ** IOMMU data structures, which may compromise security on ** [ 0.224017] ** your system. ** [ 0.227023] ** ** [ 0.229017] ** If you see this message and you are not debugging the ** [ 0.231020] ** kernel, report this immediately to your vendor! ** [ 0.234020] ** ** [ 0.237024] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.239018] ************************************************************* [ 0.241823] NET: Registered protocol family 16 [ 0.243540] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.246097] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.248081] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.252042] cpuidle: using governor menu [ 0.254427] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.256438] PCI: Using configuration type 1 for base access [ 0.258164] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.266047] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.267027] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.270237] cryptd: max_cpu_qlen set to 1000 [ 0.273288] ACPI: Added _OSI(Module Device) [ 0.275038] ACPI: Added _OSI(Processor Device) [ 0.276022] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.278017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.285234] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.293668] ACPI: Interpreter enabled [ 0.295162] ACPI: PM: (supports S0 S3 S4 S5) [ 0.296000] ACPI: Using IOAPIC for interrupt routing [ 0.299128] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.302461] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.310782] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.313067] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.315022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.317080] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.322310] acpiphp: Slot [2] registered [ 0.323112] acpiphp: Slot [5] registered [ 0.324133] acpiphp: Slot [6] registered [ 0.326100] acpiphp: Slot [7] registered [ 0.327133] acpiphp: Slot [8] registered [ 0.329180] acpiphp: Slot [9] registered [ 0.330238] acpiphp: Slot [10] registered [ 0.332237] acpiphp: Slot [3] registered [ 0.334202] acpiphp: Slot [4] registered [ 0.336208] acpiphp: Slot [11] registered [ 0.338161] acpiphp: Slot [12] registered [ 0.340166] acpiphp: Slot [13] registered [ 0.342143] acpiphp: Slot [14] registered [ 0.343145] acpiphp: Slot [15] registered [ 0.345163] acpiphp: Slot [16] registered [ 0.347203] acpiphp: Slot [17] registered [ 0.349164] acpiphp: Slot [18] registered [ 0.351157] acpiphp: Slot [19] registered [ 0.352146] acpiphp: Slot [20] registered [ 0.354138] acpiphp: Slot [21] registered [ 0.355117] acpiphp: Slot [22] registered [ 0.356172] acpiphp: Slot [23] registered [ 0.358185] acpiphp: Slot [24] registered [ 0.359141] acpiphp: Slot [25] registered [ 0.361140] acpiphp: Slot [26] registered [ 0.362105] acpiphp: Slot [27] registered [ 0.363102] acpiphp: Slot [28] registered [ 0.364088] acpiphp: Slot [29] registered [ 0.366122] acpiphp: Slot [30] registered [ 0.367094] acpiphp: Slot [31] registered [ 0.368060] PCI host bridge to bus 0000:00 [ 0.369016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.371021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.373024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.375027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.377057] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.380026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.382222] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.385054] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.388000] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.397015] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.403105] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.406021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.408022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.410020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.413860] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.417012] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.420063] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.423933] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.431023] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.446016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.450912] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.458000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.464025] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.475027] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.492020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.503631] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.523029] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.539024] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.564028] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.575574] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.588017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.594020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.613019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.631366] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.636020] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.643020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.663025] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.675651] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.684016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.691024] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.707020] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.716538] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.724026] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.731025] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.756027] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.768269] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.770333] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.772323] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.774345] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.776376] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.780106] iommu: Default domain type: Passthrough [ 0.782452] SCSI subsystem initialized [ 0.783132] ACPI: bus type USB registered [ 0.784097] usbcore: registered new interface driver usbfs [ 0.785076] usbcore: registered new interface driver hub [ 0.787068] usbcore: registered new device driver usb [ 0.788123] pps_core: LinuxPPS API ver. 1 registered [ 0.790009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.792068] PTP clock support registered [ 0.794034] EDAC MC: Ver: 3.0.0 [ 0.795118] PCI: Using ACPI for IRQ routing [ 0.796815] NetLabel: Initializing [ 0.798010] NetLabel: domain hash size = 128 [ 0.799009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.800127] NetLabel: unlabeled traffic allowed by default [ 0.802276] vgaarb: loaded [ 0.803285] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.804012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.810000] clocksource: Switched to clocksource kvm-clock [ 0.924684] VFS: Disk quotas dquot_6.6.0 [ 0.926476] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.928986] *** VALIDATE ramfs *** [ 0.930315] *** VALIDATE hugetlbfs *** [ 0.931860] pnp: PnP ACPI init [ 0.934365] pnp: PnP ACPI: found 6 devices [ 0.953674] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.956923] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.958917] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.961047] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.963905] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.966336] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.969527] NET: Registered protocol family 2 [ 0.972737] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.977860] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.981125] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.986900] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.990773] TCP: Hash tables configured (established 65536 bind 65536) [ 0.993501] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.996902] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.999860] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.002956] NET: Registered protocol family 1 [ 1.006266] RPC: Registered named UNIX socket transport module. [ 1.007911] RPC: Registered udp transport module. [ 1.009203] RPC: Registered tcp transport module. [ 1.010475] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.012099] NET: Registered protocol family 44 [ 1.013282] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.014875] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.016388] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.018179] PCI: CLS 0 bytes, default 64 [ 1.020210] Unpacking initramfs... [ 2.436557] debug: unmapping init [mem 0xffff9de97cc54000-0xffff9de97ffbffff] [ 2.441930] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.444505] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.447828] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 2.951132] Initialise system trusted keyrings [ 2.952557] Key type blacklist registered [ 2.954822] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.964481] zbud: loaded [ 2.967829] *** VALIDATE nfs *** [ 2.968827] *** VALIDATE nfs4 *** [ 2.970553] pstore: using deflate compression [ 2.976505] Platform Keyring initialized [ 3.083462] NET: Registered protocol family 38 [ 3.084808] Key type asymmetric registered [ 3.086310] Asymmetric key parser 'x509' registered [ 3.088036] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.090898] io scheduler mq-deadline registered [ 3.092680] io scheduler kyber registered [ 3.094310] io scheduler bfq registered [ 3.096110] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.099127] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.101933] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.104427] ACPI: Power Button [PWRF] [ 3.109354] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.116339] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.128505] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.136612] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.149834] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.179105] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.208914] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.214476] Non-volatile memory driver v1.3 [ 3.216270] Linux agpgart interface v0.103 [ 3.253515] virtio_blk virtio1: [vda] 145904 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.256508] vda: detected capacity change from 0 to 74702848 [ 3.270874] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.274026] vdb: detected capacity change from 0 to 1073741824 [ 3.295278] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.298619] vdc: detected capacity change from 0 to 2621440000 [ 3.320474] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.323285] vdd: detected capacity change from 0 to 2621440000 [ 3.341438] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.344588] vde: detected capacity change from 0 to 4294967296 [ 3.364081] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.367383] vdf: detected capacity change from 0 to 4294967296 [ 3.377550] libphy: Fixed MDIO Bus: probed [ 3.392698] usbcore: registered new interface driver usbserial_generic [ 3.395503] usbserial: USB Serial support registered for generic [ 3.398069] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.402541] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.404459] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.406995] mousedev: PS/2 mouse device common for all mice [ 3.410089] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.410711] rtc_cmos 00:05: RTC can wake from S4 [ 3.416994] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.418249] rtc_cmos 00:05: registered as rtc0 [ 3.425062] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.427874] intel_pstate: CPU model not supported [ 3.430621] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.434554] hid: raw HID events driver (C) Jiri Kosina [ 3.436927] usbcore: registered new interface driver usbhid [ 3.438519] usbhid: USB HID core driver [ 3.440184] drop_monitor: Initializing network drop monitor service [ 3.442252] Initializing XFRM netlink socket [ 3.443911] NET: Registered protocol family 10 [ 3.446333] Segment Routing with IPv6 [ 3.447701] NET: Registered protocol family 17 [ 3.450912] mpls_gso: MPLS GSO support [ 3.456431] RAS: Correctable Errors collector initialized. [ 3.459081] AVX version of gcm_enc/dec engaged. [ 3.460700] AES CTR mode by8 optimization enabled [ 3.536419] sched_clock: Marking stable (3536389922, 0)->(4546831764, -1010441842) [ 3.540383] registered taskstats version 1 [ 3.542750] Loading compiled-in X.509 certificates [ 3.545197] zswap: loaded using pool lzo/zbud [ 3.570302] Key type big_key registered [ 3.583860] Key type encrypted registered [ 3.586048] ima: No TPM chip found, activating TPM-bypass! [ 3.588849] ima: Allocated hash algorithm: sha1 [ 3.590993] ima: No architecture policies found [ 3.593247] evm: Initialising EVM extended attributes: [ 3.595610] evm: security.selinux [ 3.597176] evm: security.ima [ 3.598546] evm: security.capability [ 3.600266] evm: HMAC attrs: 0x1 [ 3.603035] rtc_cmos 00:05: setting system clock to 2026-08-15 21:14:26 UTC (1786828466) [ 3.608636] debug: unmapping init [mem 0xffffffff84e03000-0xffffffff84ffffff] [ 3.611236] debug: unmapping init [mem 0xffffffff83b82000-0xffffffff83e58fff] [ 3.618190] Write protecting the kernel read-only data: 28672k [ 3.621665] debug: unmapping init [mem 0xffffffff82203000-0xffffffff823fffff] [ 3.624722] debug: unmapping init [mem 0xffffffff82b14000-0xffffffff82bfffff] [ 3.657563] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.666041] systemd[1]: Detected virtualization kvm. [ 3.668280] systemd[1]: Detected architecture x86-64. [ 3.670423] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.696072] systemd[1]: No hostname configured. [ 3.697497] systemd[1]: Set hostname to . [ 3.699448] random: systemd: uninitialized urandom read (16 bytes read) [ 3.701910] systemd[1]: Initializing machine ID from random generator. [ 3.835205] random: systemd: uninitialized urandom read (16 bytes read) [ 3.837065] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.841179] random: systemd: uninitialized urandom read (16 bytes read) [ 3.843154] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.848877] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local File Systems. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. [ OK ] Reached target Sockets. Starting Journal Service... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Reached target Paths. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.548361] device-mapper: uevent: version 1.0.3 [ 4.550431] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.271265] virtio_net virtio0 ens2: renamed from eth0 [ 5.278401] random: fast init done [ 5.345451] scsi host0: ata_piix [ 5.354128] scsi host1: ata_piix [ 5.356800] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.359334] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.636875] dracut-initqueue[584]: RTNETLINK answers: File exists [ 10.105077] random: crng init done [ 10.106210] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.472201] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.609651] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.871188] SELinux: Disabled at runtime. [ 11.933956] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.943323] systemd[1]: Detected virtualization kvm. [ 11.945667] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.537473] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.543922] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.549059] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.553460] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.557366] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.565617] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.573148] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd Root File System. Starting Remount Root and Kernel File Systems... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target rpc_pipefs.target. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Apply Kernel Variables... [ 12.707237] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting POSIX Message Queue File System... [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.011091] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.360549] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.372218] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.506444] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.521653] EDAC sbridge: Ver: 1.1.2 [ 14.935292] Key type dns_resolver registered [ 15.245841] NFS: Registering the id_resolver key type [ 15.251069] Key type id_resolver registered [ 15.252377] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg460-server login: [ 42.395368] hrtimer: interrupt took 23726864 ns [ 57.346985] libcfs: loading out-of-tree module taints kernel. [ 57.457463] Key type ._llcrypt registered [ 57.459879] Key type .llcrypt registered [ 57.624759] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [ 77.576339] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 79.652168] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 79.665778] alg: No test for adler32 (adler32-zlib) [ 81.304186] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 82.143443] LNet: Added LNI 192.168.204.160@tcp [8/256/0/180] [ 83.871433] Key type lgssc registered [ 85.970984] Lustre: Echo OBD driver; http://www.lustre.org/ [ 101.885511] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 148.555621] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 161.631810] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 161.681787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 162.977069] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 163.057311] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 163.235479] Lustre: lustre-MDT0000: new disk, initializing [ 163.391088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 163.411386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 168.183032] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 183.658615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 183.873725] Lustre: 6503:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 183.909705] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 183.917614] Lustre: Skipped 1 previous similar message [ 184.090473] Lustre: lustre-MDT0001: new disk, initializing [ 184.208380] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 184.257036] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 184.273830] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 189.321883] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 194.407814] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 204.391667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 204.685818] Lustre: lustre-OST0000: new disk, initializing [ 204.689624] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 204.695633] Lustre: 8442:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 204.749372] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 205.558203] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 205.565721] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 205.601751] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 210.648137] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 224.745532] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 224.876973] Lustre: lustre-OST0001: new disk, initializing [ 224.885661] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 224.897809] Lustre: 9513:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 224.958545] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 231.161552] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 231.939752] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 231.950488] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 231.993161] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 242.291770] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 249.554363] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 256.928384] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing check_logdir /tmp/testlogs/ [ 262.139407] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing yml_node [ 266.683395] Lustre: DEBUG MARKER: Client: 2.17.57.3 [ 270.241352] Lustre: DEBUG MARKER: MDS: 2.17.57.3 [ 273.465049] Lustre: DEBUG MARKER: OSS: 2.17.57.3 [ 275.403126] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Aug 15 17:18:55 EDT 2026 [ 293.785224] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 295.617392] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 313.826131] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 313.828730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 313.859643] Lustre: Skipped 1 previous similar message [ 313.892067] Lustre: Skipped 3 previous similar messages [ 318.945124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 318.958200] Lustre: Skipped 3 previous similar messages [ 319.351521] Lustre: server umount lustre-MDT0000 complete [ 322.917198] LustreError: 6496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786828785 with bad export cookie 11145494753499178656 [ 322.924658] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 322.926569] LustreError: 6496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 324.064946] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 324.071078] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 324.078282] Lustre: Skipped 3 previous similar messages [ 324.091146] Lustre: Skipped 1 previous similar message [ 327.079857] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 327.083446] Lustre: Skipped 1 previous similar message [ 332.193928] LustreError: 8035:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 332.195506] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 332.209305] LustreError: 8035:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 332.222080] Lustre: Skipped 1 previous similar message [ 337.380315] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 337.380893] LustreError: 12457:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 337.412262] LustreError: 12457:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 337.596796] Lustre: server umount lustre-MDT0001 complete [ 340.858628] Lustre: server umount lustre-OST0000 complete [ 344.335385] Lustre: server umount lustre-OST0001 complete [ 349.741675] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [ 357.929994] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 393.765630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 393.984255] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 394.006391] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 394.068358] Lustre: lustre-MDT0000: new disk, initializing [ 394.124229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 394.135773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 400.673403] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 411.593780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 411.744907] Lustre: 15068:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 411.817671] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 411.822918] Lustre: Skipped 1 previous similar message [ 411.904705] Lustre: lustre-MDT0001: new disk, initializing [ 411.983304] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 412.037563] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 412.048668] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 416.094522] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 425.404684] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 431.806280] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 438.072547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 438.363185] Lustre: lustre-OST0000: new disk, initializing [ 438.372404] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 438.383873] Lustre: 17036:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 438.479768] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 440.054100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 440.063956] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 440.165519] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 444.327395] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 453.982969] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 458.744724] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 458.923249] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 463.264068] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 463.539991] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 465.892036] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 465.893583] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 465.925314] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 465.931261] Lustre: Skipped 3 previous similar messages [ 471.750302] Lustre: server umount lustre-OST0000 complete [ 476.132745] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 476.151779] Lustre: Skipped 1 previous similar message [ 481.978329] Lustre: server umount lustre-MDT0000 complete [ 486.300704] LustreError: 15059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786828949 with bad export cookie 11145494753499180189 [ 486.302073] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 486.312534] LustreError: 15059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 486.728971] Lustre: server umount lustre-MDT0001 complete [ 495.157230] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 17:22:34 (1786828954) [ 518.174184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 518.536762] LustreError: 19910:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 518.623927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 523.749269] LustreError: 19911:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 524.237633] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 528.882968] LustreError: 19910:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 533.324715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 537.987490] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 545.560550] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 550.999641] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 552.918645] Lustre: Failing over lustre-MDT0000 [ 552.926595] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 552.927353] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 552.936662] Lustre: Skipped 2 previous similar messages [ 552.943982] Lustre: Skipped 7 previous similar messages [ 553.225231] Lustre: server umount lustre-MDT0000 complete [ 558.052104] LustreError: 19911:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 558.078399] LustreError: 19911:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 571.649559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 571.785040] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 576.484142] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 577.507670] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 577.509779] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 577.541236] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 585.381167] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 591.854454] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 591.863499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 591.875032] Lustre: Skipped 1 previous similar message [ 591.886326] Lustre: Skipped 1 previous similar message [ 595.460613] Lustre: server umount lustre-MDT0000 complete [ 596.961060] LustreError: 19910:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 596.990509] LustreError: 19910:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 599.990061] LustreError: 19892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786829062 with bad export cookie 11145494753499181603 [ 600.003363] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 600.009318] LustreError: 19892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 600.428728] Lustre: server umount lustre-MDT0001 complete [ 612.408773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 613.417509] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 613.424052] Lustre: Skipped 2 previous similar messages [ 618.861469] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 629.006861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 635.437685] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 644.377046] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 649.882316] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 652.197947] Lustre: Failing over lustre-MDT0000 [ 652.419587] Lustre: server umount lustre-MDT0000 complete [ 654.249105] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 654.265504] LustreError: 23244:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 654.279677] LustreError: 23244:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 670.569917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 670.650868] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 670.858308] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 674.935442] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 674.944480] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 674.953290] Lustre: Skipped 1 previous similar message [ 674.980917] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 675.164727] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 685.115965] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 690.657192] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 690.683181] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 690.715680] Lustre: Skipped 1 previous similar message [ 690.743186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 695.251959] Lustre: server umount lustre-MDT0000 complete [ 699.438205] LustreError: 23229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786829162 with bad export cookie 11145494753499182541 [ 699.441908] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 699.470044] LustreError: 23229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 699.787888] Lustre: server umount lustre-MDT0001 complete [ 709.099588] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 17:26:09 (1786829169) [ 726.185717] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [ 739.691329] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 782.465513] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 796.080128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 796.475464] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 796.500920] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 796.703074] Lustre: lustre-MDT0000: new disk, initializing [ 796.840246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 796.846029] Lustre: Skipped 2 previous similar messages [ 796.864150] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 803.864828] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 819.010035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 819.188248] Lustre: 29627:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 819.484367] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 823.980689] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 833.197175] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 838.864744] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 846.047726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 846.368526] Lustre: 31598:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 847.951071] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 847.991744] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 848.089108] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 848.116120] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 852.827469] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 863.152324] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 868.471886] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 889.313619] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 889.321372] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 889.325983] Lustre: Skipped 1 previous similar message [ 889.332347] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 889.343167] Lustre: Skipped 3 previous similar messages [ 895.481211] Lustre: server umount lustre-OST0000 complete [ 904.894369] Lustre: server umount lustre-MDT0000 complete [ 909.182443] LustreError: 29621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786829372 with bad export cookie 11145494753499182947 [ 909.205615] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 909.766407] Lustre: server umount lustre-MDT0001 complete [ 920.982022] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [ 923.808730] Key type lgssc unregistered [ 924.241125] LNet: 33581:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 924.246490] LNetError: 33581:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 924.272487] LNet: Removed LNI 192.168.204.160@tcp [ 925.374873] Key type .llcrypt unregistered [ 925.378391] Key type ._llcrypt unregistered [ 954.331691] Key type ._llcrypt registered [ 954.333906] Key type .llcrypt registered [ 954.441593] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 955.389502] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 955.497607] alg: No test for adler32 (adler32-zlib) [ 956.549242] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 956.806811] LNet: Added LNI 192.168.204.160@tcp [8/256/0/180] [ 958.471236] Key type lgssc registered [ 959.415118] Lustre: Echo OBD driver; http://www.lustre.org/ [ 967.672269] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 967.691821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 969.168766] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 973.989927] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 978.401588] LustreError: 35013:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 978.432664] LustreError: 35013:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 982.307151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 982.642833] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 987.311287] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 995.455183] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1001.612558] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1008.314450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1008.578994] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1013.710950] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1014.860924] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 1022.615455] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1028.017444] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1039.100947] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 17:31:39 (1786829499) [ 1040.956917] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1042.997734] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1050.305833] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 17:31:50 (1786829510) [ 1061.170559] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 17:32:01 (1786829521) [ 1062.316857] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1071.861635] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1071.869446] Lustre: Skipped 2 previous similar messages [ 1073.951750] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 17:32:13 (1786829533) [ 1077.558380] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1077.562940] Lustre: Skipped 3 previous similar messages [ 1094.721145] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1094.727620] Lustre: Skipped 6 previous similar messages [ 1097.589333] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 17:32:37 (1786829557) [ 1100.616696] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1104.325569] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1113.139657] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 17:32:53 (1786829573) [ 1115.002546] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1115.008357] Lustre: Skipped 3 previous similar messages [ 1128.328558] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 17:33:08 (1786829588) [ 1145.828029] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1145.833857] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1145.846352] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1145.865299] Lustre: Skipped 1 previous similar message [ 1148.887595] Lustre: server umount lustre-OST0000 complete [ 1156.069463] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1156.087042] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1156.093994] Lustre: Skipped 2 previous similar messages [ 1161.190993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1161.211893] Lustre: Skipped 1 previous similar message [ 1165.283679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1165.295583] Lustre: Skipped 1 previous similar message [ 1168.351183] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1168.499410] Lustre: server umount lustre-MDT0000 complete [ 1170.406848] LustreError: 35740:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1170.420857] LustreError: 35740:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1172.936462] LustreError: 34993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786829635 with bad export cookie 5208189072919083612 [ 1172.938077] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1172.953611] LustreError: 34993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1173.447671] Lustre: server umount lustre-MDT0001 complete [ 1184.798245] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [ 1187.553233] Key type lgssc unregistered [ 1187.838323] LNet: 42880:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1187.842055] LNetError: 42880:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1187.869829] LNet: Removed LNI 192.168.204.160@tcp [ 1188.894561] Key type .llcrypt unregistered [ 1188.896333] Key type ._llcrypt unregistered [ 1206.492809] Key type ._llcrypt registered [ 1206.497923] Key type .llcrypt registered [ 1206.594122] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [ 1222.037183] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 1223.407866] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1223.443855] alg: No test for adler32 (adler32-zlib) [ 1224.588961] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 1224.841771] LNet: Added LNI 192.168.204.160@tcp [8/256/0/180] [ 1226.503322] Key type lgssc registered [ 1227.772153] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1265.053085] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1265.116682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1266.408566] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1266.468747] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1266.635822] Lustre: lustre-MDT0000: new disk, initializing [ 1266.756703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1266.782830] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1271.872526] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 1283.373146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1283.543420] Lustre: 46037:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1283.630618] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1283.634151] Lustre: Skipped 1 previous similar message [ 1283.706822] Lustre: lustre-MDT0001: new disk, initializing [ 1283.792899] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1283.830603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1283.846564] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1288.635781] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 1298.186376] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1304.187920] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1311.243199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1311.510912] Lustre: lustre-OST0000: new disk, initializing [ 1311.514890] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1311.522405] Lustre: 48004:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1311.586806] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1317.631575] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 1321.008341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1321.018095] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1321.164215] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1328.307518] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1334.371954] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1334.551496] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1340.076133] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1340.230778] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1347.049838] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1347.051407] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1347.057814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1347.096207] Lustre: Skipped 1 previous similar message [ 1352.171332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1352.177332] Lustre: Skipped 2 previous similar messages [ 1356.255183] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1356.413298] Lustre: server umount lustre-OST0000 complete [ 1360.870976] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1360.917841] LustreError: Skipped 1 previous similar message [ 1360.928563] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1360.954297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1362.404857] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1366.714664] Lustre: server umount lustre-MDT0000 complete [ 1370.150062] LustreError: 48014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786829833 with bad export cookie 3636437579000532762 [ 1370.151239] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1370.156897] LustreError: 48014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1370.532589] Lustre: server umount lustre-MDT0001 complete [ 1379.814842] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 17:37:20 (1786829840) [ 1387.575115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1388.030820] LustreError: 50466:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1388.050305] LustreError: 50466:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1388.135868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1393.122725] LustreError: 50467:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1394.236589] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 1397.235761] LustreError: 50466:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1402.339027] LustreError: 50467:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1404.106163] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1404.508950] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1409.373632] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 1417.144517] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1422.830543] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1430.506677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1430.730184] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1439.144641] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 1446.819762] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1454.754085] LustreError: 52995:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1455.463850] Lustre: Modifying parameter general.timeout=1 in log params [ 1456.815741] Lustre: Modifying parameter general.timeout=3 in log params [ 1456.818758] Lustre: Skipped 1 previous similar message [ 1459.161651] Lustre: Modifying parameter general.timeout=6 in log params [ 1459.166205] Lustre: Skipped 2 previous similar messages [ 1463.232771] Lustre: Modifying parameter general.timeout=11 in log params [ 1463.236368] Lustre: Skipped 4 previous similar messages [ 1472.346565] Lustre: Modifying parameter general.timeout=20 in log params [ 1472.349167] Lustre: Skipped 8 previous similar messages [ 1488.559952] Lustre: Modifying parameter general.timeout=35 in log params [ 1488.566326] Lustre: Skipped 14 previous similar messages [ 1521.830726] Lustre: Modifying parameter general.timeout=64 in log params [ 1521.833425] Lustre: Skipped 28 previous similar messages [ 1590.713906] Lustre: Disabling parameter general.timeout= in log params [ 1590.718238] Lustre: Skipped 65 previous similar messages [ 1593.930053] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 17:40:53 (1786830053) [ 1596.727256] Lustre: 59957:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1598.755735] Lustre: 60057:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1602.180255] Lustre: 60253:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1602.193868] Lustre: 60253:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1606.464661] Lustre: 60469:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1606.480237] Lustre: 60469:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1632.913429] Lustre: 61826:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1632.919595] LustreError: 61826:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1637.211556] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 17:41:37 (1786830097) [ 1650.149792] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1650.161743] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1650.171766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1650.178670] Lustre: Skipped 2 previous similar messages [ 1655.723574] Lustre: server umount lustre-MDT0000 complete [ 1655.780186] LustreError: 50461:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1655.800432] LustreError: 50461:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1658.986550] LustreError: 50448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786830121 with bad export cookie 3636437579000533644 [ 1658.996118] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1659.335849] Lustre: server umount lustre-MDT0001 complete [ 1673.223353] Lustre: server umount lustre-OST0000 complete [ 1697.313499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1697.519913] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1697.777054] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1702.186984] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 1711.759761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1711.864984] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1711.892273] Lustre: 64358:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1716.878445] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 1724.443273] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1729.778936] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1736.787957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1737.053536] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1737.354208] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1737.359511] Lustre: Skipped 1 previous similar message [ 1744.036019] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 1751.569171] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1753.104849] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1753.111197] Lustre: Skipped 23 previous similar messages [ 1753.135650] Lustre: 66922:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1763.815308] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1763.822522] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1763.846789] Lustre: Skipped 2 previous similar messages [ 1763.867247] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1763.879557] Lustre: Skipped 2 previous similar messages [ 1766.166562] Lustre: server umount lustre-OST0000 complete [ 1773.536272] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1773.550391] LustreError: Skipped 1 previous similar message [ 1773.553929] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1773.575547] Lustre: Skipped 1 previous similar message [ 1773.584939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1773.593809] Lustre: Skipped 2 previous similar messages [ 1784.287242] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1784.459261] Lustre: server umount lustre-MDT0000 complete [ 1784.801629] LustreError: 64368:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1784.822688] LustreError: 64368:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1787.934063] LustreError: 64350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786830250 with bad export cookie 3636437579000536780 [ 1787.945313] LustreError: 64350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1787.949628] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1788.311692] Lustre: server umount lustre-MDT0001 complete [ 1799.848627] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [ 1802.614899] Key type lgssc unregistered [ 1802.847697] LNet: 68133:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1802.855222] LNetError: 68133:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1802.866147] LNet: Removed LNI 192.168.204.160@tcp [ 1803.635651] Key type .llcrypt unregistered [ 1803.641579] Key type ._llcrypt unregistered [ 1829.584546] Key type ._llcrypt registered [ 1829.587747] Key type .llcrypt registered [ 1829.702074] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 1831.518930] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1831.610361] alg: No test for adler32 (adler32-zlib) [ 1832.967812] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 1833.564673] LNet: Added LNI 192.168.204.160@tcp [8/256/0/180] [ 1835.399277] Key type lgssc registered [ 1837.643756] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1846.747559] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1846.769703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1848.185515] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1852.312573] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 1858.536232] LustreError: 69546:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1858.555444] LustreError: 69546:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1861.124560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1861.843732] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1866.578143] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 1875.100490] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1881.176672] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1887.732342] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1888.201902] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1894.845776] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 1902.672867] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1913.022268] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1922.023794] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1922.034128] Lustre: Skipped 1 previous similar message [ 1922.036551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1927.036684] Lustre: server umount lustre-MDT0000 complete [ 1927.139561] LustreError: 69546:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1927.156561] LustreError: 69546:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1932.264752] LustreError: 69545:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1932.299769] LustreError: 69545:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1932.815453] LustreError: 69528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786830395 with bad export cookie 3768319297570781485 [ 1932.819467] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1932.834190] LustreError: 69528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1933.236091] Lustre: server umount lustre-MDT0001 complete [ 1948.062291] Lustre: server umount lustre-OST0000 complete [ 1975.356320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1975.612835] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1975.648920] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1975.870620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1979.885557] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 1988.322972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1988.421886] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1988.446181] Lustre: 74417:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1988.872180] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1993.757569] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 2002.652473] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2009.483592] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2017.134829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2017.469858] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2017.990462] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2024.466076] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 2032.883821] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2034.727227] Lustre: 76994:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2036.642735] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2044.387752] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2044.392903] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2044.408280] Lustre: Skipped 1 previous similar message [ 2044.415226] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2044.423975] Lustre: Skipped 2 previous similar messages [ 2049.406221] Lustre: server umount lustre-OST0000 complete [ 2055.136729] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2055.137232] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2055.158699] Lustre: Skipped 1 previous similar message [ 2055.170551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2055.175265] Lustre: Skipped 2 previous similar messages [ 2060.261079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2060.268896] Lustre: Skipped 1 previous similar message [ 2060.889403] Lustre: server umount lustre-MDT0000 complete [ 2064.352536] LustreError: 74428:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2064.367816] LustreError: 74428:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2064.913571] LustreError: 74409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786830527 with bad export cookie 3768319297570782311 [ 2064.926126] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2064.931689] LustreError: 74409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2065.199441] Lustre: server umount lustre-MDT0001 complete [ 2076.724422] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [ 2079.503491] Key type lgssc unregistered [ 2079.778909] LNet: 78207:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2079.788435] LNetError: 78207:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2079.807153] LNet: Removed LNI 192.168.204.160@tcp [ 2080.435220] Key type .llcrypt unregistered [ 2080.437130] Key type ._llcrypt unregistered [ 2109.138624] Key type ._llcrypt registered [ 2109.142244] Key type .llcrypt registered [ 2109.266710] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 2110.823387] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2110.904548] alg: No test for adler32 (adler32-zlib) [ 2112.662255] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 2113.014650] LNet: Added LNI 192.168.204.160@tcp [8/256/0/180] [ 2114.719203] Key type lgssc registered [ 2116.841627] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2126.313857] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2126.329511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2128.007610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2132.751475] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 2137.575146] LustreError: 79617:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2137.609252] LustreError: 79617:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2142.165650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2142.821474] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2147.770080] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 2154.305120] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2160.137757] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2166.864305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2167.627137] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2174.168828] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 2181.179742] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2186.444582] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2187.187969] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2188.858487] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2188.865384] Lustre: Skipped 1 previous similar message [ 2191.364863] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2191.369395] Lustre: Skipped 2 previous similar messages [ 2195.974050] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2195.981949] Lustre: Skipped 4 previous similar messages [ 2204.104180] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2204.110529] Lustre: Skipped 8 previous similar messages [ 2220.302718] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2220.315210] Lustre: Skipped 16 previous similar messages [ 2252.962300] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2252.973757] Lustre: Skipped 29 previous similar messages [ 2292.471657] Lustre: 87725:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2295.561347] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 17:52:35 (1786830755) [ 2297.433591] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2299.395873] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 17:52:39 (1786830759) [ 2306.017619] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2306.031102] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2306.034968] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2306.046530] Lustre: Skipped 1 previous similar message [ 2306.076207] Lustre: Skipped 1 previous similar message [ 2309.745418] Lustre: server umount lustre-OST0000 complete [ 2316.261254] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2316.272403] Lustre: Skipped 1 previous similar message [ 2316.272899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2316.289458] Lustre: Skipped 1 previous similar message [ 2320.159103] Lustre: server umount lustre-MDT0000 complete [ 2321.385239] LustreError: 79616:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2323.467194] LustreError: 79596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786830786 with bad export cookie 9241201610403642842 [ 2323.469490] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2323.480733] LustreError: 79596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2323.940738] Lustre: server umount lustre-MDT0001 complete [ 2336.265793] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [ 2338.629192] Key type lgssc unregistered [ 2338.934537] LNet: 88976:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2338.940772] LNetError: 88976:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2338.958441] LNet: Removed LNI 192.168.204.160@tcp [ 2339.685795] Key type .llcrypt unregistered [ 2339.687914] Key type ._llcrypt unregistered [ 2344.987979] Key type ._llcrypt registered [ 2344.990363] Key type .llcrypt registered [ 2345.114899] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2349.727693] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules [ 2350.992640] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2351.122525] alg: No test for adler32 (adler32-zlib) [ 2352.282034] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 2352.544228] LNet: Added LNI 192.168.204.160@tcp [8/256/0/180] [ 2354.244309] Key type lgssc registered [ 2355.870617] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2357.336510] LustreError: 90192:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2374.729709] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 2378.675156] LustreError: 90192:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2385.208994] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2385.227383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2386.874412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2391.163542] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 2396.145708] LustreError: 91204:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2396.163407] LustreError: 91204:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2401.083268] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 17:54:21 (1786830861) [ 2401.250020] LustreError: 91205:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2405.815736] LustreError: 91236:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2405.823298] LustreError: 91236:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2405.828424] LustreError: 91236:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 2406.323673] Lustre: server umount lustre-MDT0000 complete [ 2421.161606] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [ 2424.064402] Key type lgssc unregistered [ 2424.410652] LNet: 92734:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2424.415692] LNetError: 92734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2424.429931] LNet: Removed LNI 192.168.204.160@tcp [ 2425.362338] Key type .llcrypt unregistered [ 2425.364988] Key type ._llcrypt unregistered [ 2445.528277] Key type ._llcrypt registered [ 2445.530411] Key type .llcrypt registered [ 2445.641883] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 2446.922054] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2446.966235] alg: No test for adler32 (adler32-zlib) [ 2447.967620] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 2448.136894] LNet: Added LNI 192.168.204.160@tcp [8/256/0/180] [ 2449.887234] Key type lgssc registered [ 2451.243987] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2458.888393] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2458.905488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2460.385346] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2465.358533] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 2470.882442] LustreError: 93952:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2470.914453] LustreError: 93952:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2476.005909] LustreError: 93951:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2476.313726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2477.352784] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2482.644422] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 2491.280947] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2497.978651] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2505.997582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2507.150055] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2514.434451] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 2522.163983] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2583.488797] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 17:57:23 (1786831043) [ 2585.497238] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2587.958806] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 17:57:27 (1786831047) [ 2594.785032] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2594.801592] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2594.824961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2597.346810] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2597.350826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2597.381019] Lustre: Skipped 2 previous similar messages [ 2602.315507] Lustre: server umount lustre-MDT0000 complete [ 2602.468848] LustreError: 96258:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2602.499207] LustreError: 96258:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2607.207975] LustreError: 95803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786831070 with bad export cookie 10059988532472176710 [ 2607.214335] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2607.220646] LustreError: 95803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2607.586591] LustreError: 93947:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2607.593428] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2607.618079] Lustre: Skipped 1 previous similar message [ 2607.931309] Lustre: server umount lustre-MDT0001 complete [ 2614.036985] Lustre: server umount lustre-OST0000 complete [ 2627.252740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2627.582153] LustreError: 98269:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2627.610150] LustreError: 98269:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2627.772812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2633.033660] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 2637.285426] LustreError: 98270:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2637.298224] LustreError: 98270:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2641.967507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2642.964479] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2648.258623] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 2658.834933] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2667.237023] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2683.424285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2683.742975] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2683.788127] LustreError: 98258:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2683.817160] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2683.844259] LustreError: 100355:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2683.857429] LustreError: 100355:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2683.863591] LustreError: 100355:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2684.030235] Lustre: server umount lustre-OST0000 complete [ 2684.036725] LustreError: 100355:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2692.658476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2692.767946] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2692.773137] LustreError: 98259:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2692.790301] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2692.795099] LustreError: 100820:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2692.807541] LustreError: 100820:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2692.811199] LustreError: 100820:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2692.861620] Lustre: server umount lustre-OST0000 complete [ 2692.864202] LustreError: 100820:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2702.045081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2702.278650] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2702.301549] Lustre: Found index 0 for lustre-OST0000, updating log [ 2702.330537] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2702.473729] Lustre: lustre-OST0000: new disk, initializing [ 2702.479132] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2702.488418] Lustre: 101359:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2702.932977] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2709.269520] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 2713.207401] Lustre: Failing over lustre-OST0000 [ 2713.401743] Lustre: server umount lustre-OST0000 complete [ 2717.666405] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2717.667858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2717.679833] Lustre: Skipped 1 previous similar message [ 2722.786033] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2722.800291] Lustre: Skipped 2 previous similar messages [ 2723.558083] Lustre: server umount lustre-MDT0000 complete [ 2727.721649] LustreError: 98250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786831190 with bad export cookie 10059988532472179454 [ 2727.726474] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2727.733501] LustreError: 98250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2737.277323] Lustre: server umount lustre-MDT0001 complete [ 2746.982807] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 18:00:07 (1786831207) [ 2765.828426] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 2774.727908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2775.014940] LustreError: 103719:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2775.117471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2779.377816] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 2787.787168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2792.711951] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 2795.647489] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2802.354149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2802.963378] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2809.485262] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 2813.160755] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2813.207575] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2813.207836] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2818.119432] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2818.212729] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2818.300868] Lustre: lustre-OST0001: new disk, initializing [ 2818.304843] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2818.312831] Lustre: 106050:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2818.598444] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2818.603583] Lustre: Skipped 2 previous similar messages [ 2825.280319] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 2825.356657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2825.366218] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2825.516327] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2836.521057] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2848.408643] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2854.258317] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.60@tcp (stopping) [ 2855.394432] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2855.408810] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2860.101829] Lustre: server umount lustre-MDT0001 complete [ 2861.540416] LustreError: 103719:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2861.561797] LustreError: 103719:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2873.289215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2873.402632] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2873.408234] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2873.460717] Lustre: 103709:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2874.836987] LustreError: 107577:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2874.851713] LustreError: 107577:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2874.864789] LustreError: 103712:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2874.991343] LustreError: 107592:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.160@tcp: cfg command failed: rc = -17 [ 2875.001401] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2875.001401] [ 2880.000636] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2880.128702] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 2891.557428] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 18:02:31 (1786831351) [ 2912.086585] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 2933.095395] Lustre: 109670:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2958.708705] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2963.031513] Lustre: 110806:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3053.023475] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3053.033337] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3053.047503] Lustre: Skipped 2 previous similar messages [ 3053.053203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3053.062352] Lustre: Skipped 5 previous similar messages [ 3059.322292] Lustre: server umount lustre-MDT0000 complete [ 3063.174386] LustreError: 106771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786831526 with bad export cookie 10059988532472180455 [ 3063.183509] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3063.194224] LustreError: 106771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3063.549920] Lustre: server umount lustre-MDT0001 complete [ 3077.629584] Lustre: server umount lustre-OST0000 complete [ 3080.671183] Lustre: 93263:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786831527/real 1786831527] req@ffff9de8c194ad80 x1873628009717504/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786831543 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3080.739079] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3080.762254] Lustre: Skipped 3 previous similar messages [ 3081.297649] Lustre: server umount lustre-OST0001 complete [ 3087.332194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3097.531710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3105.929958] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3116.376572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3138.798857] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 3150.850199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3150.879199] Lustre: lustre-MDT0000: reset Object Index mappings [ 3151.441085] LustreError: 114753:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3151.454311] LustreError: 114753:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 3151.561030] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3151.565336] Lustre: Skipped 1 previous similar message [ 3156.800967] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 3166.506488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3166.539950] Lustre: lustre-MDT0001: reset Object Index mappings [ 3172.161690] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 3175.352573] Lustre: 115903:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3181.862628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3188.999155] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 3191.860078] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:226 to 0x280000400:257) [ 3197.703780] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3197.954578] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:354 to 0x280000401:386) [ 3203.593013] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3203.623544] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:195 to 0x2c0000400:225) [ 3205.428715] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 3213.155898] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3217.159275] Lustre: 117780:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3270.807334] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 18:08:50 (1786831730) [ 3300.320164] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3300.331789] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3300.352624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3300.359031] Lustre: Skipped 8 previous similar messages [ 3303.719086] Lustre: server umount lustre-MDT0000 complete [ 3305.970948] LustreError: 117090:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3305.999034] LustreError: 117090:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 3307.262244] LustreError: 114734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786831770 with bad export cookie 10059988532472227348 [ 3307.265593] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3307.280861] LustreError: 114734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3313.812810] Lustre: server umount lustre-MDT0001 complete [ 3324.236869] Lustre: server umount lustre-OST0000 complete [ 3342.303360] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3342.546607] Lustre: server umount lustre-OST0001 complete [ 3350.326474] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [ 3360.760190] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 3402.565615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3402.946562] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3402.990175] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3403.130972] Lustre: lustre-MDT0000: new disk, initializing [ 3403.230630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3403.233534] Lustre: Skipped 3 previous similar messages [ 3403.242943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3408.422551] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 3417.624245] Lustre: server umount lustre-MDT0000 complete [ 3427.382807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3427.645841] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3427.673458] Lustre: 122047:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3427.715064] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3431.629895] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 3436.965630] Lustre: server umount lustre-MDT0000 complete [ 3446.509891] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 18:11:46 (1786831906) [ 3448.083855] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3450.511934] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 18:11:50 (1786831910) [ 3476.069116] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 3487.405554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3492.397885] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 3502.191283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3502.391535] Lustre: 124119:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3502.474512] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3502.485484] Lustre: Skipped 1 previous similar message [ 3502.578229] Lustre: lustre-MDT0001: new disk, initializing [ 3502.791312] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3502.806339] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3508.023924] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 3514.060820] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3521.321059] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3521.709282] Lustre: lustre-OST0000: new disk, initializing [ 3521.715344] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3521.725356] Lustre: 125700:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3523.170869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3523.178893] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3523.319426] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3528.390821] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 3539.718513] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3539.881333] Lustre: lustre-OST0001: new disk, initializing [ 3539.885559] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3539.890742] Lustre: 126569:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3541.663653] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3541.676361] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3541.796839] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3546.703451] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 3557.802621] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3561.761202] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3962.747418] Lustre: 132988:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3962.753139] Lustre: 132988:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3962.757746] Lustre: 132988:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3962.762574] Lustre: 132988:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3962.768836] Lustre: 132988:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3962.775089] Lustre: 132988:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4265.571677] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 18:25:25 (1786832725) [ 4280.800678] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4280.813205] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4280.833509] Lustre: Skipped 5 previous similar messages [ 4280.839917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4280.846137] Lustre: Skipped 7 previous similar messages [ 4285.886669] Lustre: server umount lustre-MDT0000 complete [ 4289.515227] LustreError: 127385:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4289.552032] LustreError: 127385:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 4289.635823] LustreError: 127412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786832752 with bad export cookie 10059988532472296186 [ 4289.641885] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4289.649203] LustreError: 127412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4294.633308] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4294.655095] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4294.660213] Lustre: Skipped 4 previous similar messages [ 4294.668278] Lustre: Skipped 4 previous similar messages [ 4296.130273] Lustre: server umount lustre-MDT0001 complete [ 4305.965928] Lustre: server umount lustre-OST0000 complete [ 4315.980762] Lustre: server umount lustre-OST0001 complete [ 4323.100856] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [ 4337.959768] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 4383.645120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4383.903694] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4383.937653] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4384.043318] Lustre: lustre-MDT0000: new disk, initializing [ 4384.216474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4384.221765] Lustre: Skipped 5 previous similar messages [ 4384.248852] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4389.704594] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 4401.637302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4401.780765] Lustre: 136061:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4401.822207] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4401.825723] Lustre: Skipped 1 previous similar message [ 4401.901586] Lustre: lustre-MDT0001: new disk, initializing [ 4402.009963] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4402.028673] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4407.690575] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 4420.756174] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4429.456553] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4437.446440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4438.001263] Lustre: lustre-OST0000: new disk, initializing [ 4438.010879] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4438.036310] Lustre: 138029:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4438.191975] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4438.205075] Lustre: Skipped 1 previous similar message [ 4439.150602] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4439.166114] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4439.372396] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4446.767644] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 4456.786357] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4469.049697] Lustre: lustre-MDD0000: changelog on [ 4471.699686] Lustre: lustre-MDD0001: changelog on [ 6425.212464] Lustre: lustre-MDD0001: changelog off [ 6426.872595] Lustre: 138493:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6427.643424] Lustre: lustre-MDD0000: changelog off [ 6434.284953] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6434.300399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6434.302893] Lustre: Skipped 1 previous similar message [ 6438.546456] Lustre: server umount lustre-OST0000 complete [ 6444.513920] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6444.517315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6444.531582] Lustre: Skipped 1 previous similar message [ 6444.543426] Lustre: Skipped 2 previous similar messages [ 6448.054625] Lustre: server umount lustre-MDT0000 complete [ 6449.635615] LustreError: 136075:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6449.655900] LustreError: 136075:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6451.896561] LustreError: 136055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786834914 with bad export cookie 10059988532473120555 [ 6451.923833] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6452.177578] Lustre: server umount lustre-MDT0001 complete [ 6461.388533] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [ 6463.700566] Key type lgssc unregistered [ 6464.064693] LNet: 147184:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6464.074571] LNetError: 147184:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6464.097047] LNet: Removed LNI 192.168.204.160@tcp [ 6464.938243] Key type .llcrypt unregistered [ 6464.940978] Key type ._llcrypt unregistered [ 6481.201563] Key type ._llcrypt registered [ 6481.203884] Key type .llcrypt registered [ 6481.278739] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [ 6494.390627] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 6495.394849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6495.426944] alg: No test for adler32 (adler32-zlib) [ 6496.490163] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 6496.700934] LNet: Added LNI 192.168.204.160@tcp [8/256/0/180] [ 6498.343469] Key type lgssc registered [ 6499.277086] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6533.097811] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6533.127251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6534.355679] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6534.456705] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6534.589869] Lustre: lustre-MDT0000: new disk, initializing [ 6534.681967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6534.708217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6538.384867] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 6548.577194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6548.677798] Lustre: 150379:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6548.726084] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6548.732481] Lustre: Skipped 1 previous similar message [ 6548.798938] Lustre: lustre-MDT0001: new disk, initializing [ 6548.873483] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6548.900648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6548.909071] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6552.493150] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 6560.825123] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6566.258311] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6572.041410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6572.282984] Lustre: lustre-OST0000: new disk, initializing [ 6572.286360] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6572.294092] Lustre: 152348:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6572.398980] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6577.707681] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 6578.748916] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6578.760424] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6578.847728] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6585.764388] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6590.564051] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_module obdecho/obdecho [ 6592.270831] LustreError: 153351:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6592.277772] LustreError: 153351:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6593.745549] LustreError: 153448:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6604.772475] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6604.785047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6604.796276] Lustre: Skipped 2 previous similar messages [ 6604.807427] Lustre: Skipped 1 previous similar message [ 6609.898028] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6609.905659] Lustre: Skipped 3 previous similar messages [ 6610.326854] Lustre: server umount lustre-MDT0000 complete [ 6613.901121] LustreError: 150370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786835076 with bad export cookie 7598219742614635813 [ 6613.913279] LustreError: 150370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6613.919986] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6614.177492] Lustre: server umount lustre-MDT0001 complete [ 6628.253167] Lustre: server umount lustre-OST0000 complete [ 6648.415210] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [ 6657.616969] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 6688.586318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6688.797710] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6688.822880] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6688.889802] Lustre: lustre-MDT0000: new disk, initializing [ 6689.005501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6689.039069] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6693.301389] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 6702.703829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6702.841686] Lustre: 157163:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6702.905408] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6702.908915] Lustre: Skipped 1 previous similar message [ 6702.994433] Lustre: lustre-MDT0001: new disk, initializing [ 6703.083952] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6703.125283] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6703.137684] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6707.333641] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 6715.079503] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6719.626887] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6725.698829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6725.894733] Lustre: lustre-OST0000: new disk, initializing [ 6725.897821] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6725.902726] Lustre: 159131:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6725.974908] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6730.902241] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 6731.314419] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6731.326634] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6731.435100] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6738.875822] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6742.723245] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6742.902691] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6746.961497] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6747.172364] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6752.228791] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6752.231203] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6752.240375] LustreError: Skipped 1 previous similar message [ 6752.244610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6752.257155] Lustre: Skipped 1 previous similar message [ 6757.349589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6762.474567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6762.489945] Lustre: Skipped 2 previous similar messages [ 6762.975847] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6763.093350] Lustre: server umount lustre-OST0000 complete [ 6767.589588] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6767.607853] Lustre: Skipped 2 previous similar messages [ 6772.709714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6772.717080] Lustre: Skipped 3 previous similar messages [ 6773.201528] Lustre: server umount lustre-MDT0000 complete [ 6776.568735] LustreError: 157153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786835239 with bad export cookie 7598219742614636765 [ 6776.574228] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6776.583500] LustreError: 157153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6776.875244] Lustre: server umount lustre-MDT0001 complete [ 6779.912714] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 19:07:20 (1786835240) [ 6781.295336] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6782.899387] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 19:07:23 (1786835243) [ 6789.507685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6789.881199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6793.589661] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 6795.235622] LustreError: 161543:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6795.246720] LustreError: 161543:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6799.332254] LustreError: 161542:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6800.831334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6805.086101] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 6811.900732] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6816.751530] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6822.260705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6822.500477] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6822.506550] Lustre: Skipped 1 previous similar message [ 6827.228842] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 6832.847520] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6843.764791] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.60@tcp (stopping) [ 6845.409916] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6845.428286] Lustre: Skipped 1 previous similar message [ 6849.287355] Lustre: server umount lustre-MDT0000 complete [ 6850.530386] LustreError: 161543:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6850.547496] LustreError: 161543:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6852.168218] LustreError: 161522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786835315 with bad export cookie 7598219742614637654 [ 6852.174462] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6852.179774] LustreError: 161522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6853.976597] LustreError: 164036:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6855.649341] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6855.660807] Lustre: Skipped 1 previous similar message [ 6858.504436] Lustre: server umount lustre-MDT0001 complete [ 6863.956727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6874.291197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6874.456746] LustreError: 165384:0:(llog.c:1655:llog_backup()) MGC192.168.204.160@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6874.461168] Lustre: 165384:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.160@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6882.272519] LustreError: 165389:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6882.279572] LustreError: 165389:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6892.513313] LustreError: 165389:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6892.528610] LustreError: 165389:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6910.293262] LustreError: 165390:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6910.307084] LustreError: 165390:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 14 previous similar messages [ 6935.522186] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6935.528101] Lustre: 165384:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.204.160@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6943.202280] LustreError: 163373:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6943.218642] LustreError: 163373:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 22 previous similar messages [ 6976.054923] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6977.063819] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6979.532263] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 6985.766549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6997.475503] LustreError: 148602:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000d0ee55ac x1873632254093568/t0(0) o250->MGC192.168.204.160@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7001.329417] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 7003.120168] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7006.980586] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7008.176809] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7012.168592] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7013.347147] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7028.708402] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7028.710136] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7028.712030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7028.712036] Lustre: Skipped 7 previous similar messages [ 7028.725834] Lustre: Skipped 1 previous similar message [ 7029.966438] Lustre: server umount lustre-MDT0000 complete [ 7032.599793] LustreError: 161524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786835495 with bad export cookie 7598219742614644115 [ 7032.604709] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7032.611374] LustreError: 161524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7032.883403] Lustre: server umount lustre-MDT0001 complete [ 7046.198967] Lustre: server umount lustre-OST0000 complete [ 7063.156517] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [ 7070.808273] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 7100.225361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7100.366987] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7100.384558] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7100.435346] Lustre: lustre-MDT0000: new disk, initializing [ 7100.492056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7103.692923] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 7111.509285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7111.613933] Lustre: 170491:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7111.866606] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7111.879076] Lustre: Skipped 2 previous similar messages [ 7111.933445] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7115.449626] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 7122.974164] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7127.155365] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7132.316864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7132.496157] Lustre: lustre-OST0000: new disk, initializing [ 7132.498421] Lustre: Skipped 1 previous similar message [ 7132.501691] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7132.507721] Lustre: Skipped 2 previous similar messages [ 7132.513575] Lustre: 172460:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7137.506459] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 7138.375233] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7138.382019] Lustre: Skipped 1 previous similar message [ 7138.385981] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7138.502519] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7145.120224] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7148.826537] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7149.000595] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7152.557338] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7152.737125] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7159.269078] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7159.269393] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7159.274952] LustreError: Skipped 1 previous similar message [ 7159.280613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7159.280625] Lustre: Skipped 2 previous similar messages [ 7159.283992] Lustre: Skipped 2 previous similar messages [ 7160.373852] Lustre: server umount lustre-OST0000 complete [ 7169.637153] Lustre: server umount lustre-MDT0000 complete [ 7172.273744] LustreError: 170482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786835635 with bad export cookie 7598219742614660943 [ 7172.278959] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7172.282470] LustreError: 170482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7172.508046] Lustre: server umount lustre-MDT0001 complete [ 7179.366626] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 19:14:00 (1786835640) [ 7184.309806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7184.646215] LustreError: 174870:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7184.663646] LustreError: 174870:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 41 previous similar messages [ 7187.825912] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 7194.498546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7198.101658] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 7203.753180] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7207.602184] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7212.198430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7216.566867] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 7221.380218] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7244.654282] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 19:15:05 (1786835705) [ 7247.332517] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7247.336092] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7247.344959] Lustre: Skipped 3 previous similar messages [ 7253.250432] Lustre: server umount lustre-OST0000 complete [ 7256.545545] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7270.367134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7270.489281] Lustre: server umount lustre-MDT0000 complete [ 7273.018401] LustreError: 174853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786835735 with bad export cookie 7598219742614661825 [ 7273.019591] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7273.027267] LustreError: 174853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7273.286679] Lustre: server umount lustre-MDT0001 complete [ 7280.482681] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [ 7282.448718] Key type lgssc unregistered [ 7282.670914] LNet: 178622:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7282.676755] LNetError: 178622:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7283.749177] LNet: Removed LNI 192.168.204.160@tcp [ 7284.346915] Key type .llcrypt unregistered [ 7284.350981] Key type ._llcrypt unregistered [ 7297.644441] Key type ._llcrypt registered [ 7297.647342] Key type .llcrypt registered [ 7297.715436] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 7298.262689] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7298.369779] alg: No test for adler32 (adler32-zlib) [ 7299.339296] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 7299.499188] LNet: Added LNI 192.168.204.160@tcp [8/256/0/180] [ 7301.151290] Key type lgssc registered [ 7301.931282] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7306.849498] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7306.861631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7322.335520] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7327.455286] LustreError: 179843:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.160@tcp: failed processing log, type 4: rc = -110 [ 7332.575892] LustreError: 179843:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7332.587467] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7332.589898] LustreError: MGC192.168.204.160@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7332.601872] LustreError: 179843:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7332.608449] LustreError: 179843:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7332.613652] LustreError: 179843:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7332.668371] Lustre: server umount lustre-OST0000 complete [ 7332.670353] LustreError: 179843:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7338.107367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7339.352171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7341.999786] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 7347.223215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7347.494500] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7350.183447] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 7354.907745] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7358.343665] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7362.303459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7362.408273] LustreError: 182157:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7362.413763] Lustre: 182157:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.204.160@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7362.500568] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7366.322480] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 7370.529398] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7372.771182] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7372.776400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7372.780941] Lustre: Skipped 2 previous similar messages [ 7372.788233] Lustre: Skipped 2 previous similar messages [ 7377.892203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7377.902363] Lustre: Skipped 2 previous similar messages [ 7383.012643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7383.016181] Lustre: Skipped 2 previous similar messages [ 7386.079164] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7386.188774] Lustre: server umount lustre-MDT0000 complete [ 7388.130643] LustreError: 181071:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7388.148731] LustreError: 181071:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 7388.354776] LustreError: 180321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786835851 with bad export cookie 3115922998109082683 [ 7388.360531] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7388.361976] LustreError: 180321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7389.151391] LustreError: 181073:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000007e4c8f01 x1873633096001152/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7389.290699] Lustre: server umount lustre-MDT0001 complete [ 7401.992899] Lustre: server umount lustre-OST0000 complete [ 7407.930162] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [ 7414.134520] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 7438.233681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7438.456660] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7438.497566] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7438.593289] Lustre: lustre-MDT0000: new disk, initializing [ 7438.667802] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7438.686759] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7441.879079] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 7449.198092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7449.272960] Lustre: 185688:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7449.311903] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7449.317258] Lustre: Skipped 1 previous similar message [ 7449.373505] Lustre: lustre-MDT0001: new disk, initializing [ 7449.444433] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7449.483373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7449.491510] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7452.330879] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 7459.521135] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7463.197878] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7467.729571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7467.914195] Lustre: lustre-OST0000: new disk, initializing [ 7467.917810] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7467.921303] Lustre: 187656:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7467.973850] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7471.738178] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 7476.749944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7476.760850] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7476.826084] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7477.817546] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7480.709103] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7480.848490] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7483.726646] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7483.857665] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7486.955592] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7486.960285] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7486.961430] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7486.964554] Lustre: Skipped 1 previous similar message [ 7486.983872] Lustre: Skipped 1 previous similar message [ 7491.208974] Lustre: server umount lustre-OST0000 complete [ 7495.649887] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7495.659105] LustreError: Skipped 1 previous similar message [ 7495.661788] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7495.673634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7499.596745] Lustre: server umount lustre-MDT0000 complete [ 7502.138853] LustreError: 186484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786835965 with bad export cookie 3115922998109083390 [ 7502.141269] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7502.148533] LustreError: 186484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7502.372772] Lustre: server umount lustre-MDT0001 complete [ 7508.865231] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 19:19:29 (1786835969) [ 7520.930866] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [ 7522.896633] Key type lgssc unregistered [ 7523.151696] LNet: 190600:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7523.168278] LNetError: 190600:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7524.202826] LNet: Removed LNI 192.168.204.160@tcp [ 7524.912519] Key type .llcrypt unregistered [ 7524.914450] Key type ._llcrypt unregistered [ 7539.811703] Key type ._llcrypt registered [ 7539.814079] Key type .llcrypt registered [ 7539.882746] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 7540.643472] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7540.655730] alg: No test for adler32 (adler32-zlib) [ 7541.614359] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 7541.806316] LNet: Added LNI 192.168.204.160@tcp [8/256/0/180] [ 7543.423178] Key type lgssc registered [ 7544.271305] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7549.726184] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7549.744052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7551.039812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7553.909486] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 7559.799641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7560.090436] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7563.008286] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 7568.253498] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7572.130078] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7577.443322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7577.596122] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7577.598567] LustreError: 193680:0:(llog.c:1696:llog_backup()) MGC192.168.204.160@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7577.604128] Lustre: 193680:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.204.160@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7577.685248] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7582.001994] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 7587.189986] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7590.370424] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7590.373964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7590.382932] Lustre: Skipped 2 previous similar messages [ 7590.385743] Lustre: Skipped 1 previous similar message [ 7594.727588] Lustre: server umount lustre-MDT0000 complete [ 7595.490283] LustreError: 191818:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7595.503659] LustreError: 191818:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 7597.232695] LustreError: 191797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786836060 with bad export cookie 14366117080008355721 [ 7597.237645] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7597.246780] LustreError: 191797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7597.412465] Lustre: server umount lustre-MDT0001 complete [ 7610.179664] Lustre: server umount lustre-OST0000 complete [ 7615.316802] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [ 7621.554635] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 7645.774248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7645.922279] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7645.939876] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7645.988990] Lustre: lustre-MDT0000: new disk, initializing [ 7646.044643] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7646.058624] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7649.091567] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 7656.379931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7656.447980] Lustre: 197200:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7656.482610] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7656.484686] Lustre: Skipped 1 previous similar message [ 7656.487307] LustreError: 197983:0:(llog.c:1696:llog_backup()) MGC192.168.204.160@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7656.492549] LustreError: 197983:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7656.500457] Lustre: 197983:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.160@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7656.512160] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7656.515690] Lustre: Skipped 1 previous similar message [ 7656.559657] Lustre: lustre-MDT0001: new disk, initializing [ 7656.622187] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7656.652319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7656.660293] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7659.647210] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 7666.270101] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7669.949956] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7674.405405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7674.559717] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7674.561713] Lustre: Skipped 1 previous similar message [ 7674.563649] LustreError: 199143:0:(llog.c:1696:llog_backup()) MGC192.168.204.160@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7674.570428] LustreError: 199143:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7674.574798] Lustre: 199143:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.160@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7674.580535] Lustre: 199143:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7674.594518] Lustre: lustre-OST0000: new disk, initializing [ 7674.597472] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7674.601346] Lustre: 199168:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7674.657807] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7678.583979] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 7683.605250] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7683.610425] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7683.671819] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7684.585143] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7687.394361] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7687.544497] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7690.122328] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7690.259740] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7693.794816] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7693.797111] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7693.808403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7693.823559] Lustre: Skipped 1 previous similar message [ 7697.395120] Lustre: server umount lustre-OST0000 complete [ 7703.011456] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7703.017000] LustreError: Skipped 1 previous similar message [ 7703.020402] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7703.030456] Lustre: Skipped 1 previous similar message [ 7703.036062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7705.943813] Lustre: server umount lustre-MDT0000 complete [ 7708.325809] LustreError: 197190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786836171 with bad export cookie 14366117080008356435 [ 7708.329886] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7708.334171] LustreError: 197190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7708.542558] Lustre: server umount lustre-MDT0001 complete [ 7713.937127] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 19:22:54 (1786836174) [ 7723.198509] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 7729.127300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7729.330777] LustreError: 202291:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7729.356615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7732.008772] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 7734.752258] LustreError: 202292:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7737.487368] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7740.563391] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 7742.515397] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7746.715961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7750.619590] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 7755.564255] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7755.702343] Lustre: lustre-OST0001: new disk, initializing [ 7755.707207] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7755.711579] Lustre: 204598:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7759.561274] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 7760.881817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7760.889938] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7760.910772] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7767.175443] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7772.242857] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7773.403622] Lustre: DEBUG MARKER: ADD OST3 [ 7776.378708] loop: module loaded [ 7776.414139] loop0: detected capacity change from 0 to 204800000 [ 7776.561752] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7776.569431] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7776.576276] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7780.513429] loop0: detected capacity change from 0 to 204800000 [ 7782.481368] Lustre: DEBUG MARKER: STOP OST3 [ 7784.307448] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7784.444531] Lustre: lustre-OST0003: new disk, initializing [ 7784.447486] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7784.451783] Lustre: 206524:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7784.545036] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7784.550562] Lustre: Skipped 3 previous similar messages [ 7785.745667] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7785.862139] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7785.869233] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7785.902967] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7786.112876] Lustre: Failing over lustre-OST0003 [ 7786.198827] Lustre: server umount lustre-OST0003 complete [ 7786.350091] LustreError: 203775:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.204.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7787.807078] LustreError: 206699:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7788.719834] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 7791.087503] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7791.092913] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7791.101308] Lustre: Skipped 2 previous similar messages [ 7791.104809] LustreError: 203773:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7791.116414] LustreError: 203773:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7795.768228] Lustre: DEBUG MARKER: START OST3 again [ 7799.982939] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7800.166286] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7801.638689] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7801.652509] Lustre: lustre-OST0003: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 7801.654535] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7804.115388] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 7820.292327] Lustre: Failing over lustre-OST0003 [ 7820.353845] Lustre: server umount lustre-OST0003 complete [ 7820.781451] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7820.782956] LustreError: 203773:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7820.788355] Lustre: Skipped 1 previous similar message [ 7820.798351] LustreError: 203773:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 7835.104657] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7835.109644] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7835.118294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7835.120623] Lustre: Skipped 2 previous similar messages [ 7840.692599] Lustre: server umount lustre-MDT0000 complete [ 7841.250941] LustreError: 203018:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7841.267748] LustreError: 203018:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 7842.682293] LustreError: 203026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786836305 with bad export cookie 14366117080008357317 [ 7842.683699] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7842.689805] LustreError: 203026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7846.370924] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7846.377789] Lustre: Skipped 5 previous similar messages [ 7848.935472] Lustre: server umount lustre-MDT0001 complete [ 7851.099316] Lustre: server umount lustre-OST0000 complete [ 7853.174124] Lustre: server umount lustre-OST0001 complete [ 7856.545989] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [ 7861.351552] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 7884.363194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7884.523978] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7884.545122] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7884.598220] Lustre: lustre-MDT0000: new disk, initializing [ 7884.667050] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7884.671634] Lustre: Skipped 1 previous similar message [ 7884.681845] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7887.007762] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 7893.444189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7893.544239] Lustre: 212052:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7893.740365] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7896.293632] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 7902.248806] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7905.816667] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7909.968941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7910.217412] Lustre: lustre-OST0000: new disk, initializing [ 7910.221706] Lustre: Skipped 1 previous similar message [ 7910.227872] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7910.230916] Lustre: Skipped 2 previous similar messages [ 7910.234424] Lustre: 214019:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7911.767554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7911.774141] Lustre: Skipped 1 previous similar message [ 7911.779293] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7911.851656] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7914.019129] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 7919.884609] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7922.601282] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7922.714677] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7925.047851] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7925.189625] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7927.265914] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7927.271865] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7927.276586] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7927.278346] Lustre: Skipped 6 previous similar messages [ 7927.284321] Lustre: Skipped 3 previous similar messages [ 7932.453923] Lustre: server umount lustre-OST0000 complete [ 7940.748353] Lustre: server umount lustre-MDT0000 complete [ 7942.625376] LustreError: 212063:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7942.634123] LustreError: 212063:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 7943.031300] LustreError: 212043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786836405 with bad export cookie 14366117080008359599 [ 7943.041463] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7943.042514] LustreError: 212043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7943.227736] Lustre: server umount lustre-MDT0001 complete [ 7945.491484] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 19:26:46 (1786836406) [ 7955.430568] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [ 7961.783916] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 7984.709897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7984.869395] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7984.888815] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7984.964269] Lustre: lustre-MDT0000: new disk, initializing [ 7985.039159] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7987.897282] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 7994.486208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7994.574147] Lustre: 218625:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7994.725084] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7997.963697] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8004.193204] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8007.423412] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8011.264508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8011.411593] Lustre: 220592:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8012.724395] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8014.966402] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8020.892119] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8023.642597] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8023.776023] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8026.813423] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8026.961675] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8033.250096] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8033.252555] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8033.254137] LustreError: Skipped 3 previous similar messages [ 8033.266517] Lustre: Skipped 2 previous similar messages [ 8033.276408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8033.286453] Lustre: Skipped 6 previous similar messages [ 8034.627687] Lustre: server umount lustre-OST0000 complete [ 8043.160910] Lustre: server umount lustre-MDT0000 complete [ 8043.489671] LustreError: 219855:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8043.499887] LustreError: 219855:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8045.570513] LustreError: 220591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786836508 with bad export cookie 14366117080008360488 [ 8045.578639] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8045.581303] LustreError: 220591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8045.793569] Lustre: server umount lustre-MDT0001 complete [ 8051.162955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8051.474768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8051.477603] Lustre: Skipped 5 previous similar messages [ 8054.361779] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8059.523938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8062.111411] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8066.369107] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8069.749556] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8073.793034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8077.424934] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8081.482575] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8089.571246] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8089.573928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8089.582709] Lustre: Skipped 3 previous similar messages [ 8091.252783] Lustre: server umount lustre-OST0000 complete [ 8093.593737] Lustre: server umount lustre-MDT0000 complete [ 8095.805898] LustreError: 222787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786836558 with bad export cookie 14366117080008361363 [ 8095.810880] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8095.812782] LustreError: 222787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8096.011887] Lustre: server umount lustre-MDT0001 complete [ 8218.640897] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [ 8220.248574] Key type lgssc unregistered [ 8220.432750] LNet: 226471:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8220.436276] LNetError: 226471:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8221.477629] LNet: Removed LNI 192.168.204.160@tcp [ 8222.004134] Key type .llcrypt unregistered [ 8222.006972] Key type ._llcrypt unregistered [ 8234.896149] Key type ._llcrypt registered [ 8234.897753] Key type .llcrypt registered [ 8234.967891] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [ 8243.638668] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 8244.293591] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8244.303064] alg: No test for adler32 (adler32-zlib) [ 8245.258337] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 8245.431315] LNet: Added LNI 192.168.204.160@tcp [8/256/0/180] [ 8247.031230] Key type lgssc registered [ 8247.615768] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8264.839609] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8264.848584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8265.964328] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8265.979914] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8266.021742] Lustre: lustre-MDT0000: new disk, initializing [ 8266.063196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8266.073191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8267.911904] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8273.540444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8273.604807] Lustre: 229796:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8273.643910] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8273.646460] Lustre: Skipped 1 previous similar message [ 8273.686906] Lustre: lustre-MDT0001: new disk, initializing [ 8273.749892] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8273.776214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8273.782083] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8275.828787] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8280.408627] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8282.953147] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8286.057096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8286.185278] Lustre: lustre-OST0000: new disk, initializing [ 8286.188131] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8286.191883] Lustre: 231762:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8286.234903] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8288.804670] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8293.354937] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8293.896133] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8293.903243] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8293.961283] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8295.307661] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8295.417765] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8297.178380] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8297.270279] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8298.979507] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8298.980368] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8298.984329] LustreError: Skipped 1 previous similar message [ 8298.989782] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8298.993845] Lustre: Skipped 1 previous similar message [ 8300.195484] Lustre: server umount lustre-OST0000 complete [ 8304.097122] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8304.097907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8304.102917] Lustre: Skipped 2 previous similar messages [ 8309.218750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8309.222747] Lustre: Skipped 2 previous similar messages [ 8314.336947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8314.342562] Lustre: Skipped 1 previous similar message [ 8316.383161] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8316.485206] Lustre: server umount lustre-MDT0000 complete [ 8318.146900] LustreError: 229790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786836781 with bad export cookie 6262749077882827568 [ 8318.152648] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8318.154849] LustreError: 229790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8318.310790] Lustre: server umount lustre-MDT0001 complete [ 8327.087787] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 8331.738459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8331.955477] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8333.791723] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8337.378609] LustreError: 234692:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8337.387267] LustreError: 234692:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8337.861183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8338.044001] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8339.919917] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8341.338334] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8344.302421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8347.127206] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8350.861725] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8350.943231] Lustre: lustre-OST0001: new disk, initializing [ 8350.945883] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8350.949532] Lustre: 237001:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8350.997697] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8351.003936] Lustre: Skipped 1 previous similar message [ 8353.113982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8353.120196] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8353.180517] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8353.699895] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8359.953695] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8364.037546] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8370.217467] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 19:33:51 (1786836831) [ 8371.686411] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8371.692385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8371.695667] Lustre: Skipped 3 previous similar messages [ 8371.703439] Lustre: Skipped 3 previous similar messages [ 8373.445985] Lustre: server umount lustre-MDT0000 complete [ 8375.409130] LustreError: 234672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786836838 with bad export cookie 6262749077882828450 [ 8375.411765] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8375.417046] LustreError: 234672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8375.618241] Lustre: server umount lustre-MDT0001 complete [ 8388.122624] Lustre: server umount lustre-OST0000 complete [ 8400.638618] Lustre: server umount lustre-OST0001 complete [ 8404.235246] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [ 8409.589918] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 8428.891595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8429.004918] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8429.022449] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8429.067108] Lustre: lustre-MDT0000: new disk, initializing [ 8429.105586] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8429.122639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8431.064789] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8436.993436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8437.053622] Lustre: 240944:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8437.094581] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8437.097562] Lustre: Skipped 1 previous similar message [ 8437.143385] Lustre: lustre-MDT0001: new disk, initializing [ 8437.214028] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8437.219983] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8439.411903] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8444.841792] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8448.013351] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8451.408644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8451.552390] Lustre: 242912:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8453.013965] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8453.089211] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8454.565594] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8459.733451] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8461.574228] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8461.662561] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8463.362414] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8463.459228] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8468.448128] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8468.451651] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8468.452114] LustreError: Skipped 1 previous similar message [ 8468.461418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8468.465752] Lustre: Skipped 1 previous similar message [ 8470.303334] Lustre: server umount lustre-OST0000 complete [ 8473.056644] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8473.059663] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8473.069555] Lustre: Skipped 1 previous similar message [ 8486.367121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8486.450237] Lustre: server umount lustre-MDT0000 complete [ 8488.196580] LustreError: 240935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786836951 with bad export cookie 6262749077882830557 [ 8488.200161] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8488.204094] LustreError: 240935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8488.345821] Lustre: server umount lustre-MDT0001 complete [ 8492.091253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8492.378048] LustreError: 245125:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8492.416907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8492.419643] Lustre: Skipped 2 previous similar messages [ 8494.380633] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8497.632584] LustreError: 245126:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8498.042199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8500.005195] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8503.544893] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8506.137974] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8509.646332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8513.090294] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8517.196817] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8666.592415] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8666.594876] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8666.597277] Lustre: Skipped 2 previous similar messages [ 8666.599430] Lustre: Skipped 6 previous similar messages [ 8676.320642] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8676.399748] Lustre: server umount lustre-OST0000 complete [ 8684.384270] Lustre: server umount lustre-MDT0000 complete [ 8686.431057] LustreError: 245107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786837149 with bad export cookie 6262749077882831439 [ 8686.433660] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8686.439067] LustreError: 245107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8686.583206] Lustre: server umount lustre-MDT0001 complete [ 8783.626263] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [ 8785.025302] Key type lgssc unregistered [ 8785.190299] LNet: 248733:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8785.195868] LNetError: 248733:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8785.207962] LNet: Removed LNI 192.168.204.160@tcp [ 8785.630170] Key type .llcrypt unregistered [ 8785.633076] Key type ._llcrypt unregistered [ 8797.054289] Key type ._llcrypt registered [ 8797.055949] Key type .llcrypt registered [ 8797.101292] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [ 8804.964138] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 8805.501744] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8805.511289] alg: No test for adler32 (adler32-zlib) [ 8806.414318] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 8806.527801] LNet: Added LNI 192.168.204.160@tcp [8/256/0/180] [ 8808.135241] Key type lgssc registered [ 8808.609865] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8827.228107] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 8831.535335] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8831.544061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8832.678667] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8832.699985] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8832.740310] Lustre: lustre-MDT0000: new disk, initializing [ 8832.780183] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8832.789145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8834.648903] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8839.530883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8839.580353] Lustre: 252775:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8839.626050] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8839.627684] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8839.630813] Lustre: Skipped 1 previous similar message [ 8839.670945] Lustre: lustre-MDT0001: new disk, initializing [ 8839.712414] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8839.727514] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8839.731769] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8841.382051] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8843.952485] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8846.640377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8846.762296] Lustre: lustre-OST0000: new disk, initializing [ 8846.764709] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8846.768567] Lustre: 254408:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8846.811428] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8849.226120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8849.231587] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8849.298405] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8849.406285] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8854.572206] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8854.657138] Lustre: lustre-OST0001: new disk, initializing [ 8854.659826] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8854.663319] Lustre: 255280:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8854.705259] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8856.853560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8856.861564] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8856.917285] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8857.159372] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8863.321432] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8869.656794] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8875.488099] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8875.491216] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8875.499243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8877.544548] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8877.546560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8877.550918] Lustre: Skipped 2 previous similar messages [ 8877.554120] Lustre: Skipped 1 previous similar message [ 8880.183403] Lustre: server umount lustre-MDT0000 complete [ 8881.752313] LustreError: 252768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786837344 with bad export cookie 192971878423056613 [ 8881.754657] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8881.757841] LustreError: 252768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8881.902126] Lustre: server umount lustre-MDT0001 complete [ 8893.907335] Lustre: server umount lustre-OST0000 complete [ 8906.024138] Lustre: server umount lustre-OST0001 complete [ 8911.953739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8915.977843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8919.537629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 8923.311542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 8930.710468] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 8934.600212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8934.611635] Lustre: lustre-MDT0000: reset Object Index mappings [ 8934.805093] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8936.412272] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8939.728331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8939.739567] Lustre: lustre-MDT0001: reset Object Index mappings [ 8941.484118] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8942.585326] Lustre: 261421:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8945.177177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8945.316638] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8945.320641] Lustre: Skipped 1 previous similar message [ 8947.564336] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8949.411542] LustreError: 261774:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8949.418909] LustreError: 261774:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8951.116708] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8953.642715] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8959.469213] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8959.469675] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 8962.712813] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8964.312105] Lustre: 263288:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8969.696681] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8969.701231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8969.702223] Lustre: Skipped 1 previous similar message [ 8969.704141] Lustre: Skipped 2 previous similar messages [ 8972.220864] Lustre: server umount lustre-MDT0000 complete [ 8973.831360] LustreError: 260259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786837436 with bad export cookie 192971878423063375 [ 8973.836267] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8973.836925] LustreError: 260259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8973.979640] Lustre: server umount lustre-MDT0001 complete [ 8986.059921] Lustre: server umount lustre-OST0000 complete [ 8997.930249] Lustre: server umount lustre-OST0001 complete [ 9005.908389] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 19:44:27 (1786837467) [ 9012.820194] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [ 9017.034359] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 9031.568646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9031.676091] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9031.690281] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9031.731118] Lustre: lustre-MDT0000: new disk, initializing [ 9031.764260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9031.766775] Lustre: Skipped 1 previous similar message [ 9031.774127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9033.438214] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9038.738425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9038.789751] Lustre: 267538:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9038.934724] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9040.777427] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9045.277984] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9047.565074] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9050.588523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9050.718762] Lustre: lustre-OST0000: new disk, initializing [ 9050.720786] Lustre: Skipped 1 previous similar message [ 9050.723347] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9050.725991] Lustre: Skipped 2 previous similar messages [ 9050.728212] Lustre: 269504:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9052.163443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9052.166707] Lustre: Skipped 1 previous similar message [ 9052.168983] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9052.216188] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9053.172968] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9057.356742] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9058.955589] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9059.041287] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9060.677929] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9060.764019] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9062.371887] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9062.373169] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9062.374526] LustreError: Skipped 1 previous similar message [ 9062.375221] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9062.378800] Lustre: Skipped 3 previous similar messages [ 9062.380382] Lustre: Skipped 3 previous similar messages [ 9067.693348] Lustre: server umount lustre-OST0000 complete [ 9069.540436] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9069.545930] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9072.608298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9072.610991] Lustre: Skipped 5 previous similar messages [ 9075.570073] Lustre: server umount lustre-MDT0000 complete [ 9077.091933] LustreError: 267530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786837539 with bad export cookie 192971878423065461 [ 9077.094801] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9077.098660] LustreError: 267530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9077.236749] Lustre: server umount lustre-MDT0001 complete [ 9084.314378] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 9088.089057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9088.234733] LustreError: 272430:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9088.248337] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9088.250112] Lustre: Skipped 2 previous similar messages [ 9089.769195] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9093.046557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9094.820859] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9096.131136] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9098.755443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9101.015237] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9104.317204] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9104.384873] Lustre: lustre-OST0001: new disk, initializing [ 9104.387343] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9104.390298] Lustre: 274736:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9105.802896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9105.806566] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9105.853707] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9106.606530] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9111.657740] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9113.275489] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9115.369594] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 9115.390827] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 9118.687820] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9118.690762] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9118.694863] Lustre: Skipped 1 previous similar message [ 9118.696937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9123.815689] Lustre: server umount lustre-MDT0000 complete [ 9125.359981] LustreError: 272413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786837588 with bad export cookie 192971878423066343 [ 9125.365070] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9125.365484] LustreError: 272413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9125.512625] Lustre: server umount lustre-MDT0001 complete [ 9136.202230] Lustre: server umount lustre-OST0000 complete [ 9147.809720] Lustre: server umount lustre-OST0001 complete [ 9154.342443] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 9158.112607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9158.268170] LustreError: 277786:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9158.292853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9158.295806] Lustre: Skipped 3 previous similar messages [ 9159.796600] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9163.277060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9165.071546] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9166.373862] Lustre: 278928:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9169.032035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9171.720189] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9173.285885] LustreError: 279283:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9175.096792] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9177.711921] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9183.722443] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 9183.730868] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 9186.852941] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9188.187723] Lustre: 280802:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9190.309298] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 9190.318505] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 9193.950608] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 19:47:35 (1786837655) [ 9209.315394] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9209.320175] Lustre: Skipped 5 previous similar messages [ 9209.321037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9209.324497] Lustre: Skipped 7 previous similar messages [ 9211.680746] Lustre: server umount lustre-MDT0000 complete [ 9213.062543] LustreError: 277769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786837675 with bad export cookie 192971878423072657 [ 9213.067200] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9213.067556] LustreError: 277769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9214.432437] LustreError: 277783:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9214.439415] LustreError: 277783:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9219.247129] Lustre: server umount lustre-MDT0001 complete [ 9226.690809] Lustre: server umount lustre-OST0000 complete [ 9234.076907] Lustre: server umount lustre-OST0001 complete [ 9236.427976] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [ 9239.995903] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 9255.455091] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 9258.957464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9259.043016] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9259.053419] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9259.087499] Lustre: lustre-MDT0000: new disk, initializing [ 9259.111228] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9260.418612] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9264.401845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9264.442104] Lustre: 285037:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9264.544036] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9266.009198] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9270.771455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9270.893626] Lustre: 286668:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9271.340878] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9273.016096] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9277.329661] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9279.079547] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9279.518228] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9284.058362] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9289.394612] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 19:49:10 (1786837750) [ 9290.207927] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9290.210669] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9290.214021] Lustre: Skipped 3 previous similar messages [ 9290.216410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9290.218329] Lustre: Skipped 4 previous similar messages [ 9296.222519] Lustre: server umount lustre-MDT0000 complete [ 9297.405184] LustreError: 285029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786837760 with bad export cookie 192971878423080539 [ 9297.408792] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9297.410536] LustreError: 285029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9297.523592] Lustre: server umount lustre-MDT0001 complete [ 9308.608963] Lustre: server umount lustre-OST0000 complete [ 9320.028534] Lustre: server umount lustre-OST0001 complete [ 9322.846039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9323.004593] LustreError: 289780:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9323.011814] LustreError: 289780:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9323.031994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9323.034027] Lustre: Skipped 7 previous similar messages [ 9324.323722] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9327.111379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9328.626280] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9330.998934] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9332.820098] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9335.035124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9337.116539] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9339.609405] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9342.437120] LustreError: 291608:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9342.442988] LustreError: 291608:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9356.153553] loop1: detected capacity change from 0 to 204800000 [ 9356.218771] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9356.222823] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9356.228306] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9357.903338] loop1: detected capacity change from 0 to 204800000 [ 9359.844326] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9359.880692] LustreError: 289770:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs13091: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 9359.885841] LustreError: 289770:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs13091 [ 9359.887920] LustreError: 289770:0:(mgs_handler.c:517:mgs_target_reg()) fs13091-MDT0001: Error checking target index: rc = -13 [ 9359.891906] LustreError: fs13091-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 9359.895688] LustreError: 293513:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 9359.899187] LustreError: 293513:0:(tgt_mount.c:2129:server_put_super()) no obd fs13091-MDT0001 [ 9359.902203] LustreError: 293513:0:(tgt_mount.c:116:server_deregister_mount()) fs13091-MDT0001 not registered [ 9359.915217] Lustre: server umount fs13091-MDT0001 complete [ 9359.916845] LustreError: 293513:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 9362.684200] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9362.713649] Lustre: Setting parameter fs13091-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs13091-MDT0001 [ 9362.718116] Lustre: Skipped 2 previous similar messages [ 9362.736133] Lustre: srv-fs13091-MDT0001: No data found on store. Initialize space. [ 9362.738547] Lustre: Skipped 4 previous similar messages [ 9362.777360] Lustre: fs13091-MDT0001: new disk, initializing [ 9362.779232] Lustre: Skipped 3 previous similar messages [ 9362.813639] Lustre: srv-fs13091-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 9362.816077] Lustre: 293979:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs13091-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9362.818757] LustreError: 293979:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs13091-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9364.164018] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9367.011993] Lustre: server umount fs13091-MDT0001 complete [ 9371.489630] loop1: detected capacity change from 0 to 204800000 [ 9373.316496] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9373.429612] Lustre: 295543:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs13091-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9373.432197] LustreError: 295543:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs13091-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9374.704492] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9375.572904] LustreError: 291984:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9375.578153] LustreError: 291984:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 17 previous similar messages [ 9376.225967] Lustre: server umount fs13091-MDT0001 complete [ 9383.394630] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9383.399019] Lustre: Skipped 4 previous similar messages [ 9385.624883] Lustre: server umount lustre-MDT0000 complete [ 9388.751623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9388.806477] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9390.374200] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9392.916469] loop1: detected capacity change from 0 to 204800000 [ 9394.149848] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9394.861135] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9394.974190] Lustre: 297984:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs13091-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9394.977596] LustreError: 297984:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs13091-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9396.279818] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9397.926816] Lustre: server umount fs13091-MDT0001 complete [ 9408.991816] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9419.231075] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9419.289104] Lustre: server umount lustre-OST0000 complete [ 9424.865074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9424.866929] Lustre: Skipped 14 previous similar messages [ 9426.778937] Lustre: server umount lustre-MDT0000 complete [ 9427.993431] LustreError: 289761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786837890 with bad export cookie 192971878423083213 [ 9427.995906] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9427.997871] LustreError: 289761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9428.116778] Lustre: server umount lustre-MDT0001 complete [ 9431.744157] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [ 9432.728204] Key type lgssc unregistered [ 9432.867485] LNet: 300328:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9432.870817] LNetError: 300328:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9432.883598] LNet: Removed LNI 192.168.204.160@tcp [ 9433.191127] Key type .llcrypt unregistered [ 9433.192350] Key type ._llcrypt unregistered [ 9441.065141] Key type ._llcrypt registered [ 9441.066232] Key type .llcrypt registered [ 9441.105681] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 9441.470443] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9441.532505] alg: No test for adler32 (adler32-zlib) [ 9442.397663] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 9442.485362] LNet: Added LNI 192.168.204.160@tcp [8/256/0/180] [ 9444.071241] Key type lgssc registered [ 9444.525162] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9447.301080] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9447.306126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9448.455743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9450.019887] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9453.102681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9453.241817] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9454.783798] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9457.244442] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9459.030961] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9461.476398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9461.598706] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9463.615329] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9463.653066] LustreError: 303407:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9463.659971] LustreError: 303407:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9466.142332] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9468.898466] LustreError: 303404:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9474.020352] LustreError: 303774:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9474.029167] LustreError: 303774:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9476.444967] LustreError: 303406:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9481.558029] LustreError: 303407:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9481.564768] LustreError: 303407:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9482.833953] loop1: detected capacity change from 0 to 204800000 [ 9482.917983] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9482.922576] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9482.928174] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9484.723386] loop1: detected capacity change from 0 to 204800000 [ 9486.708798] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9486.761141] Lustre: fs19326-OST0000: new disk, initializing [ 9486.763740] Lustre: srv-fs19326-OST0000: No data found on store. Initialize space. [ 9486.767036] Lustre: 305329:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9486.797139] Lustre: fs19326-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9488.836967] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9491.798306] LustreError: 303407:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9491.805301] LustreError: 303407:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 9492.550240] Lustre: server umount fs19326-OST0000 complete [ 9496.870784] loop1: detected capacity change from 0 to 204800000 [ 9498.790267] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9498.848163] Lustre: fs19326-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9500.827921] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9503.803199] loop2: detected capacity change from 0 to 204800000 [ 9503.882081] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9503.886722] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9503.891255] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9505.730533] loop2: detected capacity change from 0 to 204800000 [ 9507.694024] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9507.727469] LustreError: fs19326-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 9507.732428] LustreError: fs19326-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 9507.735563] LustreError: 308367:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 9507.738599] LustreError: 308367:0:(tgt_mount.c:2129:server_put_super()) no obd fs19326-OST0001 [ 9507.741142] LustreError: 308367:0:(tgt_mount.c:116:server_deregister_mount()) fs19326-OST0001 not registered [ 9507.753381] Lustre: server umount fs19326-OST0001 complete [ 9507.754879] LustreError: 308367:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 9509.860616] LustreError: 303774:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9509.867944] LustreError: 303774:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 9511.364404] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9511.408448] Lustre: fs19326-OST0001: new disk, initializing [ 9511.410516] Lustre: srv-fs19326-OST0001: No data found on store. Initialize space. [ 9511.413277] Lustre: 308917:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9511.438905] Lustre: fs19326-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9513.429487] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9517.255994] Lustre: server umount fs19326-OST0000 complete [ 9520.788850] Lustre: server umount fs19326-OST0001 complete [ 9525.290732] loop1: detected capacity change from 0 to 204800000 [ 9527.275167] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9529.516329] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9532.040325] loop2: detected capacity change from 0 to 204800000 [ 9534.186695] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9534.283055] Lustre: fs19326-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9534.287638] Lustre: Skipped 1 previous similar message [ 9536.972290] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9540.668656] Lustre: server umount fs19326-OST0000 complete [ 9542.996734] LustreError: 303407:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9543.003778] LustreError: 303407:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 18 previous similar messages [ 9544.314542] Lustre: server umount fs19326-OST0001 complete [ 9550.818917] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9550.820082] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9550.821928] LustreError: Skipped 1 previous similar message [ 9550.826934] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9550.830157] Lustre: Skipped 1 previous similar message [ 9554.714473] Lustre: server umount lustre-OST0000 complete [ 9561.056550] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9561.056988] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9561.057506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9561.066927] Lustre: Skipped 2 previous similar messages [ 9562.205660] Lustre: server umount lustre-MDT0000 complete [ 9563.474089] LustreError: 301560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786838026 with bad export cookie 14087168837574088628 [ 9563.477270] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9563.478652] LustreError: 301560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9563.602570] Lustre: server umount lustre-MDT0001 complete [ 9567.421759] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [ 9568.523158] Key type lgssc unregistered [ 9568.666393] LNet: 314828:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9568.669223] LNetError: 314828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9568.679370] LNet: Removed LNI 192.168.204.160@tcp [ 9569.020111] Key type .llcrypt unregistered [ 9569.021482] Key type ._llcrypt unregistered [ 9580.186210] Key type ._llcrypt registered [ 9580.187463] Key type .llcrypt registered [ 9580.222829] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [ 9586.322302] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 9586.729849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9586.766417] alg: No test for adler32 (adler32-zlib) [ 9587.630918] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 9587.722600] LNet: Added LNI 192.168.204.160@tcp [8/256/0/180] [ 9589.311159] Key type lgssc registered [ 9589.740446] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9602.051303] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9602.058317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9603.150071] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9603.161773] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9603.195909] Lustre: lustre-MDT0000: new disk, initializing [ 9603.218993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9603.224281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9604.656175] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9609.046055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9609.096208] Lustre: 318655:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9609.122229] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9609.126017] Lustre: Skipped 1 previous similar message [ 9609.164716] Lustre: lustre-MDT0001: new disk, initializing [ 9609.204102] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9609.220758] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9609.227459] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9610.667649] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9614.539774] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9616.418937] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9618.871679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9618.977952] Lustre: lustre-OST0000: new disk, initializing [ 9618.979793] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9618.982947] Lustre: 320625:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9619.017680] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9619.389141] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9619.394347] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9619.440112] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9621.140136] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9624.971157] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9625.708584] Lustre: Mounted lustre-client - version 2.17.57_3_g928f38d [ 9626.057401] Lustre: Unmounted lustre-client [ 9629.666166] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9629.666384] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9629.669577] LustreError: Skipped 1 previous similar message [ 9629.674381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9629.678470] Lustre: Skipped 1 previous similar message [ 9631.201715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9631.203967] Lustre: Skipped 1 previous similar message [ 9635.282859] Lustre: server umount lustre-OST0000 complete [ 9639.904030] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9639.906745] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9639.910834] Lustre: Skipped 1 previous similar message [ 9639.912869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9641.440759] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9642.838213] Lustre: server umount lustre-MDT0000 complete [ 9644.029109] LustreError: 320624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786838106 with bad export cookie 8504663827050546049 [ 9644.032188] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9644.032913] LustreError: 320624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9644.139415] Lustre: server umount lustre-MDT0001 complete [ 9647.986903] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [ 9649.158101] Key type lgssc unregistered [ 9680.031478] LNet: 322786:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9680.034883] LNetError: 322786:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9757.861331] LNet: Removed LNI 192.168.204.160@tcp [ 9758.168421] Key type .llcrypt unregistered [ 9758.169988] Key type ._llcrypt unregistered [ 9763.997386] Key type ._llcrypt registered [ 9763.998441] Key type .llcrypt registered [ 9764.037191] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [ 9770.635532] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 9770.963733] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9771.020349] alg: No test for adler32 (adler32-zlib) [ 9771.879150] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 9771.973612] LNet: Added LNI 192.168.204.160@tcp [8/256/0/180] [ 9773.559083] Key type lgssc registered [ 9773.942516] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9784.924840] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 19:57:26 (1786838246) [ 9787.439709] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9787.446397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9788.532340] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9788.544887] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9788.580187] Lustre: lustre-MDT0000: new disk, initializing [ 9788.605810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9788.613085] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9790.073346] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9794.492788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9794.544067] Lustre: 325982:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9794.571369] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9794.573794] Lustre: Skipped 1 previous similar message [ 9794.608809] Lustre: lustre-MDT0001: new disk, initializing [ 9794.645717] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9794.666895] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9794.673273] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9796.142748] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9800.005233] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9801.880246] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9804.156252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9804.270348] Lustre: lustre-OST0000: new disk, initializing [ 9804.272780] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9804.276890] Lustre: 327952:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9804.312307] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9806.272403] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9809.920210] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9810.426621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9810.430062] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9810.457081] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9822.939364] Lustre: server umount lustre-OST0000 complete [ 9825.247678] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9825.250190] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9825.254244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9825.761847] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9825.768398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9830.880189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9830.882664] Lustre: Skipped 1 previous similar message [ 9835.002378] Lustre: server umount lustre-MDT0000 complete [ 9836.220370] LustreError: 325976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786838299 with bad export cookie 2145037408157907184 [ 9836.222668] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9836.224345] LustreError: 325976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9846.804625] Lustre: server umount lustre-MDT0001 complete [ 9850.470476] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [ 9851.529500] Key type lgssc unregistered [ 9851.651521] LNet: 329863:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9851.653678] LNetError: 329863:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9851.663348] LNet: Removed LNI 192.168.204.160@tcp [ 9851.949723] Key type .llcrypt unregistered [ 9851.950912] Key type ._llcrypt unregistered [ 9859.771121] Key type ._llcrypt registered [ 9859.772075] Key type .llcrypt registered [ 9859.804966] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 9860.158261] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9860.197907] alg: No test for adler32 (adler32-zlib) [ 9861.052199] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 9861.136454] LNet: Added LNI 192.168.204.160@tcp [8/256/0/180] [ 9862.719198] Key type lgssc registered [ 9863.070868] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9865.602932] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9865.610243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9866.737858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9868.066480] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9870.949429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9871.072986] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9872.408553] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9874.708634] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9876.514173] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9878.669168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9878.779695] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9880.651859] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9883.000256] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9896.845480] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9897.369840] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9898.842705] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.60@tcp (stopping) [ 9900.816307] Lustre: server umount lustre-MDT0000 complete [ 9902.047962] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9902.050475] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9902.058360] LustreError: 331113:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9902.063676] LustreError: 331113:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9902.073307] LustreError: 331094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786838364 with bad export cookie 12052068238061937168 [ 9902.077543] LustreError: 331094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9902.077696] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9902.560280] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9902.560824] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9902.565895] Lustre: Skipped 1 previous similar message [ 9902.570542] LustreError: 333602:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9903.956045] LustreError: 331137:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9903.965369] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.60@tcp (stopping) [ 9904.228615] Lustre: server umount lustre-MDT0001 complete [ 9909.083913] Lustre: lustre-OST0000: Not available for connect from 192.168.204.60@tcp (stopping) [ 9911.579699] Lustre: server umount lustre-OST0000 complete [ 9914.599279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9916.002175] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9916.998363] Lustre: 335308:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.160@tcp [ 9917.753908] Lustre: server umount MGS complete [ 9920.886154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9921.035605] LustreError: 335878:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9921.057421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9922.683480] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9926.112065] LustreError: 335879:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9926.116836] LustreError: 335879:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9926.315723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9926.478945] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9928.351565] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9934.355573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9934.631068] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9939.715232] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9946.102493] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9950.735781] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 9955.425148] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9956.949854] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9962.471303] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9962.478187] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9962.499715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9967.925783] Lustre: server umount lustre-OST0000 complete [ 9971.680262] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9971.682110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9971.690165] Lustre: Skipped 2 previous similar messages [ 9971.698127] Lustre: Skipped 4 previous similar messages [ 9976.410461] Lustre: server umount lustre-MDT0000 complete [ 9976.799935] LustreError: 337024:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9976.806367] LustreError: 337024:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9977.977495] LustreError: 335860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786838440 with bad export cookie 12052068238061937994 [ 9977.981117] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9977.984340] LustreError: 335860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9978.105799] Lustre: server umount lustre-MDT0001 complete [ 9982.488876] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [ 9983.682671] Key type lgssc unregistered [ 9983.814401] LNet: 339315:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9983.818682] LNetError: 339315:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9983.827255] LNet: Removed LNI 192.168.204.160@tcp [ 9984.206070] Key type .llcrypt unregistered [ 9984.207390] Key type ._llcrypt unregistered [ 9991.819138] Key type ._llcrypt registered [ 9991.821157] Key type .llcrypt registered [ 9991.870922] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [ 9999.923615] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [10000.419750] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10000.438592] alg: No test for adler32 (adler32-zlib) [10001.375238] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [10001.531579] LNet: Added LNI 192.168.204.160@tcp [8/256/0/180] [10003.151179] Key type lgssc registered [10003.848871] Lustre: Echo OBD driver; http://www.lustre.org/ [10020.051137] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10020.063315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10021.179583] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10021.194769] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10021.235738] Lustre: lustre-MDT0000: new disk, initializing [10021.268311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10021.277215] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10023.019657] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [10028.261737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10028.311210] Lustre: 342460:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10028.346886] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10028.349157] Lustre: Skipped 1 previous similar message [10028.388648] Lustre: lustre-MDT0001: new disk, initializing [10028.428424] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10028.445923] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10028.450518] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10030.066864] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [10034.820545] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10037.170651] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10039.884080] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10040.003437] Lustre: lustre-OST0000: new disk, initializing [10040.006113] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10040.010766] Lustre: 344427:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10040.051590] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10042.439638] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [10043.775966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10043.780348] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10043.824586] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10047.193947] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10049.113899] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10049.217278] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10051.223249] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10051.335922] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10054.114923] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10054.115141] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10054.118375] LustreError: Skipped 1 previous similar message [10054.119298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10054.124219] Lustre: Skipped 1 previous similar message [10054.135442] Lustre: Skipped 1 previous similar message [10058.432948] Lustre: server umount lustre-OST0000 complete [10064.352574] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10064.354513] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10064.361800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10064.366185] Lustre: Skipped 1 previous similar message [10066.531848] Lustre: server umount lustre-MDT0000 complete [10068.358798] LustreError: 342453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786838531 with bad export cookie 14520170093074197699 [10068.361041] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10068.365751] LustreError: 342453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10068.543445] Lustre: server umount lustre-MDT0001 complete [10072.532442] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 20:02:13 (1786838533) [10079.926230] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [10084.458134] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [10101.376425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10101.492835] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10101.509351] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10101.556387] Lustre: lustre-MDT0000: new disk, initializing [10101.608662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10101.622207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10103.696239] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [10109.004119] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10109.053850] Lustre: 349176:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10109.085301] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10109.090526] Lustre: Skipped 1 previous similar message [10109.136331] Lustre: lustre-MDT0001: new disk, initializing [10109.185509] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10109.203219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10109.207483] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10110.930657] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [10115.984082] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10118.730397] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10121.940915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10122.058375] Lustre: lustre-OST0000: new disk, initializing [10122.060929] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10122.065158] Lustre: 351148:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10122.112642] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10124.787286] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [10127.371970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10127.377760] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10127.421896] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10130.020149] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10147.810095] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10147.810393] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10147.814098] LustreError: Skipped 1 previous similar message [10147.821135] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10147.825074] Lustre: Skipped 1 previous similar message [10151.401596] Lustre: server umount lustre-OST0000 complete [10155.488649] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10155.492348] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10155.497746] Lustre: Skipped 1 previous similar message [10155.501525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10159.299824] Lustre: server umount lustre-MDT0000 complete [10161.096892] LustreError: 351147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786838623 with bad export cookie 14520170093074198581 [10161.100469] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10161.104236] LustreError: 351147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10161.253133] Lustre: server umount lustre-MDT0001 complete [10166.219222] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [10167.471908] Key type lgssc unregistered [10167.634663] LNet: 353300:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10167.640350] LNetError: 353300:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10167.653753] LNet: Removed LNI 192.168.204.160@tcp [10168.034438] Key type .llcrypt unregistered [10168.036383] Key type ._llcrypt unregistered [10210.244885] Key type ._llcrypt registered [10210.246510] Key type .llcrypt registered [10210.295724] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [10210.431621] Key type .llcrypt unregistered [10210.433368] Key type ._llcrypt unregistered [10247.187989] Key type ._llcrypt registered [10247.190290] Key type .llcrypt registered [10247.246139] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [10247.425536] Key type .llcrypt unregistered [10247.429170] Key type ._llcrypt unregistered [10262.682771] Key type ._llcrypt registered [10262.684493] Key type .llcrypt registered [10262.737490] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [10262.906319] Key type .llcrypt unregistered [10262.908493] Key type ._llcrypt unregistered [10277.994815] Key type ._llcrypt registered [10277.996133] Key type .llcrypt registered [10278.042499] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [10278.188557] Key type .llcrypt unregistered [10278.190037] Key type ._llcrypt unregistered [10313.107324] Key type ._llcrypt registered [10313.109162] Key type .llcrypt registered [10313.155331] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [10313.305863] Key type .llcrypt unregistered [10313.308367] Key type ._llcrypt unregistered [10327.264235] Key type ._llcrypt registered [10327.265792] Key type .llcrypt registered [10327.312604] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [10327.441410] Key type .llcrypt unregistered [10327.443244] Key type ._llcrypt unregistered [10340.644492] Key type ._llcrypt registered [10340.646088] Key type .llcrypt registered [10340.693866] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [10340.818422] Key type .llcrypt unregistered [10340.820145] Key type ._llcrypt unregistered [10354.257982] Key type ._llcrypt registered [10354.259081] Key type .llcrypt registered [10354.309747] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [10354.448066] Key type .llcrypt unregistered [10354.449886] Key type ._llcrypt unregistered [10367.729285] Key type ._llcrypt registered [10367.731373] Key type .llcrypt registered [10367.773104] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [10367.901516] Key type .llcrypt unregistered [10367.903307] Key type ._llcrypt unregistered [10380.640307] Key type ._llcrypt registered [10380.641831] Key type .llcrypt registered [10380.698040] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [10380.850481] Key type .llcrypt unregistered [10380.853130] Key type ._llcrypt unregistered [10407.749601] Key type ._llcrypt registered [10407.751539] Key type .llcrypt registered [10407.798454] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [10407.975418] Key type .llcrypt unregistered [10407.976728] Key type ._llcrypt unregistered [10421.190127] Key type ._llcrypt registered [10421.191553] Key type .llcrypt registered [10421.244665] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [10421.397505] Key type .llcrypt unregistered [10421.399546] Key type ._llcrypt unregistered [10434.261027] Key type ._llcrypt registered [10434.262349] Key type .llcrypt registered [10434.302574] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [10434.414339] Key type .llcrypt unregistered [10434.415964] Key type ._llcrypt unregistered [10454.292658] Key type ._llcrypt registered [10454.293883] Key type .llcrypt registered [10454.332588] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [10461.506870] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [10461.948034] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10461.984399] alg: No test for adler32 (adler32-zlib) [10462.847209] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [10462.945297] LNet: Added LNI 192.168.204.160@tcp [8/256/0/180] [10464.535158] Key type lgssc registered [10465.001163] Lustre: Echo OBD driver; http://www.lustre.org/ [10477.854925] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10477.861540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10478.954195] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [10478.966810] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [10479.004085] Lustre: tst250fs-MDT0000: new disk, initializing [10479.033919] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10479.040573] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10480.519604] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [10485.248906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10485.302577] Lustre: 373278:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10485.332788] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [10485.335521] Lustre: Skipped 1 previous similar message [10485.377955] Lustre: tst250fs-MDT0001: new disk, initializing [10485.420025] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10485.435099] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10485.438823] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10487.054373] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [10489.516526] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10492.037709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10492.146876] Lustre: tst250fs-OST0000: new disk, initializing [10492.149387] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [10492.153366] Lustre: 374910:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10492.192898] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10494.407996] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [10498.042230] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10498.046117] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10498.103373] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10498.881335] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10498.946895] Lustre: tst250fs-OST0001: new disk, initializing [10498.948962] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [10498.951909] Lustre: 375784:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10498.992165] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10501.157762] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [10503.882941] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [10506.563880] Lustre: 373283:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.204.160@tcp: IR log tst250fs-mdtir failed, not fatal: rc = -2 [10516.448508] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10516.451288] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10516.455792] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [10519.520786] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [10519.522349] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10519.531394] Lustre: Skipped 1 previous similar message [10524.640808] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [10524.643039] Lustre: Skipped 3 previous similar messages [10529.247203] Lustre: tst250fs-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10529.330708] Lustre: server umount tst250fs-MDT0000 complete [10530.690612] LustreError: 374075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786838993 with bad export cookie 6681334918822089979 [10530.696858] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10530.811389] Lustre: server umount tst250fs-MDT0001 complete [10543.110775] Lustre: server umount tst250fs-OST0000 complete [10554.781474] Lustre: server umount tst250fs-OST0001 complete [10561.603996] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [10565.102924] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [10578.177234] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 20:10:39 (1786839039) [10578.896270] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [10585.495602] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [10589.631095] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [10601.792658] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 10325 sec ========= 20:11:02 (1786839062) [10602.421675] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 20:11:03 (1786839063) === [10603.742659] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 20:11:04 (1786839064) === [10618.656134] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [10619.814325] Key type lgssc unregistered [10619.948405] LNet: 384890:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10619.951549] LNetError: 384890:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10619.962521] LNet: Removed LNI 192.168.204.160@tcp [10620.311159] Key type .llcrypt unregistered [10620.312743] Key type ._llcrypt unregistered