[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 487088948 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003351] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.011054] LSM: Security Framework initializing [ 0.012000] Yama: becoming mindful. [ 0.012043] SELinux: Initializing. [ 0.013000] *** VALIDATE selinux *** [ 0.021189] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025000] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025177] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026116] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028068] *** VALIDATE tmpfs *** [ 0.029453] *** VALIDATE proc *** [ 0.030251] *** VALIDATE cgroup *** [ 0.031013] *** VALIDATE cgroup2 *** [ 0.032267] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033150] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034013] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035031] Spectre V2 : User space: Vulnerable [ 0.036010] Speculative Store Bypass: Vulnerable [ 0.039225] debug: unmapping init [mem 0xffffffff94259000-0xffffffff94260fff] [ 0.041931] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042732] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043025] ... version: 2 [ 0.044012] ... bit width: 48 [ 0.045012] ... generic registers: 4 [ 0.046016] ... value mask: 0000ffffffffffff [ 0.047016] ... max period: 00007fffffffffff [ 0.048013] ... fixed-purpose events: 3 [ 0.049012] ... event mask: 000000070000000f [ 0.050299] rcu: Hierarchical SRCU implementation. [ 0.052453] smp: Bringing up secondary CPUs ... [ 0.053601] x86: Booting SMP configuration: [ 0.054027] .... node #0, CPUs: #1 #2 #3 [ 0.057594] smp: Brought up 1 node, 4 CPUs [ 0.059015] smpboot: Max logical packages: 1 [ 0.060022] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.227176] node 0 deferred pages initialised in 165ms [ 0.230447] devtmpfs: initialized [ 0.232270] x86/mm: Memory block size: 128MB [ 0.235961] gcov: version magic: 0x41383552 [ 0.238324] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.242107] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.245309] pinctrl core: initialized pinctrl subsystem [ 0.247177] [ 0.247784] ************************************************************* [ 0.250017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.253017] ** ** [ 0.255019] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.257017] ** ** [ 0.259014] ** This means that this kernel is built to expose internal ** [ 0.262017] ** IOMMU data structures, which may compromise security on ** [ 0.265015] ** your system. ** [ 0.266014] ** ** [ 0.268014] ** If you see this message and you are not debugging the ** [ 0.270013] ** kernel, report this immediately to your vendor! ** [ 0.272014] ** ** [ 0.274014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.276026] ************************************************************* [ 0.277681] NET: Registered protocol family 16 [ 0.279431] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.282116] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.284061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.286430] cpuidle: using governor menu [ 0.287531] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.289358] PCI: Using configuration type 1 for base access [ 0.291126] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.300133] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.304052] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.308101] cryptd: max_cpu_qlen set to 1000 [ 0.312310] ACPI: Added _OSI(Module Device) [ 0.313022] ACPI: Added _OSI(Processor Device) [ 0.314017] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.315021] ACPI: Added _OSI(Processor Aggregator Device) [ 0.320206] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.329479] ACPI: Interpreter enabled [ 0.331088] ACPI: PM: (supports S0 S3 S4 S5) [ 0.333020] ACPI: Using IOAPIC for interrupt routing [ 0.335131] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.340502] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.351115] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.353062] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.356025] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.360100] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.367408] acpiphp: Slot [2] registered [ 0.369200] acpiphp: Slot [5] registered [ 0.371195] acpiphp: Slot [6] registered [ 0.373127] acpiphp: Slot [7] registered [ 0.375139] acpiphp: Slot [8] registered [ 0.376116] acpiphp: Slot [9] registered [ 0.378126] acpiphp: Slot [10] registered [ 0.380114] acpiphp: Slot [3] registered [ 0.381113] acpiphp: Slot [4] registered [ 0.382096] acpiphp: Slot [11] registered [ 0.384097] acpiphp: Slot [12] registered [ 0.386239] acpiphp: Slot [13] registered [ 0.389154] acpiphp: Slot [14] registered [ 0.390130] acpiphp: Slot [15] registered [ 0.392129] acpiphp: Slot [16] registered [ 0.393138] acpiphp: Slot [17] registered [ 0.395151] acpiphp: Slot [18] registered [ 0.396148] acpiphp: Slot [19] registered [ 0.398166] acpiphp: Slot [20] registered [ 0.400139] acpiphp: Slot [21] registered [ 0.402147] acpiphp: Slot [22] registered [ 0.404154] acpiphp: Slot [23] registered [ 0.405148] acpiphp: Slot [24] registered [ 0.407151] acpiphp: Slot [25] registered [ 0.409139] acpiphp: Slot [26] registered [ 0.411178] acpiphp: Slot [27] registered [ 0.413136] acpiphp: Slot [28] registered [ 0.414186] acpiphp: Slot [29] registered [ 0.416126] acpiphp: Slot [30] registered [ 0.418130] acpiphp: Slot [31] registered [ 0.420098] PCI host bridge to bus 0000:00 [ 0.422032] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.424034] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.427034] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.430110] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.433043] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.436060] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.439287] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.442035] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.445512] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.456855] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.462049] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.465036] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.467019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.471023] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.474306] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.476789] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.479052] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.482902] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.489021] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.503033] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.510971] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.517536] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.529020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.536019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.573026] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.595152] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.613077] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.622027] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.641021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.652016] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.662022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.668021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.693021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.704174] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.712020] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.721022] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.742017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.751295] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.759022] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.768021] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.789021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.802605] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.809022] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.817063] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.834024] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.847473] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.849416] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.851387] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.854421] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.856236] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.860201] iommu: Default domain type: Passthrough [ 0.863460] SCSI subsystem initialized [ 0.864166] ACPI: bus type USB registered [ 0.866120] usbcore: registered new interface driver usbfs [ 0.868116] usbcore: registered new interface driver hub [ 0.871091] usbcore: registered new device driver usb [ 0.873224] pps_core: LinuxPPS API ver. 1 registered [ 0.875016] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.880150] PTP clock support registered [ 0.886155] EDAC MC: Ver: 3.0.0 [ 0.888192] PCI: Using ACPI for IRQ routing [ 0.890194] NetLabel: Initializing [ 0.891015] NetLabel: domain hash size = 128 [ 0.892016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.895107] NetLabel: unlabeled traffic allowed by default [ 0.897385] vgaarb: loaded [ 0.899282] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.901014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.908411] clocksource: Switched to clocksource kvm-clock [ 1.020472] VFS: Disk quotas dquot_6.6.0 [ 1.023880] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.026913] *** VALIDATE ramfs *** [ 1.028131] *** VALIDATE hugetlbfs *** [ 1.029830] pnp: PnP ACPI init [ 1.032437] pnp: PnP ACPI: found 6 devices [ 1.048673] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.052750] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.055565] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.057847] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.060426] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.062917] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.065985] NET: Registered protocol family 2 [ 1.068415] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.073117] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.076911] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.082517] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.086273] TCP: Hash tables configured (established 65536 bind 65536) [ 1.089502] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.092714] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.096140] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.099195] NET: Registered protocol family 1 [ 1.102089] RPC: Registered named UNIX socket transport module. [ 1.104703] RPC: Registered udp transport module. [ 1.106017] RPC: Registered tcp transport module. [ 1.107852] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.110320] NET: Registered protocol family 44 [ 1.111854] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.113940] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.116334] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.118679] PCI: CLS 0 bytes, default 64 [ 1.120445] Unpacking initramfs... [ 2.523946] debug: unmapping init [mem 0xffff8db0fcc54000-0xffff8db0fffbffff] [ 2.530435] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.532809] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.535817] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.038624] Initialise system trusted keyrings [ 3.041117] Key type blacklist registered [ 3.046241] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.055986] zbud: loaded [ 3.059143] *** VALIDATE nfs *** [ 3.060113] *** VALIDATE nfs4 *** [ 3.061513] pstore: using deflate compression [ 3.065064] Platform Keyring initialized [ 3.177445] NET: Registered protocol family 38 [ 3.179399] Key type asymmetric registered [ 3.181084] Asymmetric key parser 'x509' registered [ 3.183139] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.186257] io scheduler mq-deadline registered [ 3.187493] io scheduler kyber registered [ 3.189274] io scheduler bfq registered [ 3.191309] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.194086] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.197108] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.199947] ACPI: Power Button [PWRF] [ 3.205138] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.211836] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.238236] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.247400] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.264152] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.294113] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.321897] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.328233] Non-volatile memory driver v1.3 [ 3.329744] Linux agpgart interface v0.103 [ 3.367860] virtio_blk virtio1: [vda] 145416 512-byte logical blocks (74.5 MB/71.0 MiB) [ 3.370876] vda: detected capacity change from 0 to 74452992 [ 3.385189] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.388364] vdb: detected capacity change from 0 to 1073741824 [ 3.403260] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.406783] vdc: detected capacity change from 0 to 2621440000 [ 3.423647] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.426082] vdd: detected capacity change from 0 to 2621440000 [ 3.444847] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.447375] vde: detected capacity change from 0 to 4294967296 [ 3.462482] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.465358] vdf: detected capacity change from 0 to 4294967296 [ 3.476440] libphy: Fixed MDIO Bus: probed [ 3.481566] usbcore: registered new interface driver usbserial_generic [ 3.483899] usbserial: USB Serial support registered for generic [ 3.485972] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.489812] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.491437] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.493884] mousedev: PS/2 mouse device common for all mice [ 3.497301] rtc_cmos 00:05: RTC can wake from S4 [ 3.500279] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.504668] rtc_cmos 00:05: registered as rtc0 [ 3.506864] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.512883] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.513612] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.519768] intel_pstate: CPU model not supported [ 3.524411] hid: raw HID events driver (C) Jiri Kosina [ 3.526994] usbcore: registered new interface driver usbhid [ 3.529032] usbhid: USB HID core driver [ 3.531900] drop_monitor: Initializing network drop monitor service [ 3.534362] Initializing XFRM netlink socket [ 3.538022] NET: Registered protocol family 10 [ 3.542338] Segment Routing with IPv6 [ 3.543704] NET: Registered protocol family 17 [ 3.546352] mpls_gso: MPLS GSO support [ 3.556385] RAS: Correctable Errors collector initialized. [ 3.558646] AVX version of gcm_enc/dec engaged. [ 3.560289] AES CTR mode by8 optimization enabled [ 3.636686] sched_clock: Marking stable (3636661406, 0)->(4528427957, -891766551) [ 3.639789] registered taskstats version 1 [ 3.641465] Loading compiled-in X.509 certificates [ 3.643986] zswap: loaded using pool lzo/zbud [ 3.670544] Key type big_key registered [ 3.683358] Key type encrypted registered [ 3.684507] ima: No TPM chip found, activating TPM-bypass! [ 3.685853] ima: Allocated hash algorithm: sha1 [ 3.687021] ima: No architecture policies found [ 3.688223] evm: Initialising EVM extended attributes: [ 3.689547] evm: security.selinux [ 3.690417] evm: security.ima [ 3.691110] evm: security.capability [ 3.692079] evm: HMAC attrs: 0x1 [ 3.694118] rtc_cmos 00:05: setting system clock to 2026-08-15 18:34:41 UTC (1786818881) [ 3.699440] debug: unmapping init [mem 0xffffffff95203000-0xffffffff953fffff] [ 3.702134] debug: unmapping init [mem 0xffffffff93f82000-0xffffffff94258fff] [ 3.710122] Write protecting the kernel read-only data: 28672k [ 3.713448] debug: unmapping init [mem 0xffffffff92603000-0xffffffff927fffff] [ 3.716217] debug: unmapping init [mem 0xffffffff92f14000-0xffffffff92ffffff] [ 3.747733] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.755294] systemd[1]: Detected virtualization kvm. [ 3.757212] systemd[1]: Detected architecture x86-64. [ 3.759041] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.784487] systemd[1]: No hostname configured. [ 3.786650] systemd[1]: Set hostname to . [ 3.789136] random: systemd: uninitialized urandom read (16 bytes read) [ 3.792262] systemd[1]: Initializing machine ID from random generator. [ 3.834781] random: ln: uninitialized urandom read (6 bytes read) [ 3.934641] random: systemd: uninitialized urandom read (16 bytes read) [ 3.938523] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.943623] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.947979] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.634734] device-mapper: uevent: version 1.0.3 [ 4.636572] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.383527] virtio_net virtio0 ens2: renamed from eth0 [ 5.443770] random: fast init done [ 5.478410] scsi host0: ata_piix [ 5.526885] scsi host1: ata_piix [ 5.534978] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.537747] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.214404] dracut-initqueue[580]: RTNETLINK answers: File exists [ 10.197190] random: crng init done [ 10.199485] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.674243] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.733992] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.997674] SELinux: Disabled at runtime. [ 12.058142] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.066992] systemd[1]: Detected virtualization kvm. [ 12.069048] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.557852] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.563069] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.567462] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.573335] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.577716] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.585940] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.590921] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Switch Root. Starting Remount Root and Kernel File Systems... Mounting Huge Pages File System... [ OK ] Reached target Slices. Mounting Kernel Debug File System... [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. Activating swap /dev/disk/by-label/SWAP... Mounting POSIX Message Queue File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Create list of required st…ce nodes for the [ 12.692337] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS current kernel... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target rpc_pipefs.target. Starting Apply Kernel Variables... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 13.003442] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.265476] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.321759] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.434875] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.454231] EDAC sbridge: Ver: 1.1.2 [ 15.166629] Key type dns_resolver registered [ 15.470444] NFS: Registering the id_resolver key type [ 15.472334] Key type id_resolver registered [ 15.474162] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg657-server login: [ 42.699095] hrtimer: interrupt took 3943721 ns [ 51.356527] libcfs: loading out-of-tree module taints kernel. [ 51.402144] Key type ._llcrypt registered [ 51.405226] Key type .llcrypt registered [ 51.478785] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 71.073843] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 72.887974] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 72.907307] alg: No test for adler32 (adler32-zlib) [ 74.411189] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [ 75.398781] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 77.176138] Key type lgssc registered [ 79.024147] Lustre: Echo OBD driver; http://www.lustre.org/ [ 94.399963] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 133.447708] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 145.764242] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 145.799600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 147.042864] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 147.074310] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 147.185453] Lustre: lustre-MDT0000: new disk, initializing [ 147.263422] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 147.279981] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 151.004663] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 162.780481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 162.901648] Lustre: 6507:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 162.924440] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 162.929286] Lustre: Skipped 1 previous similar message [ 162.998124] Lustre: lustre-MDT0001: new disk, initializing [ 163.061986] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 163.083641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 163.096833] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 166.758551] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 170.874445] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 179.021769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 179.288717] Lustre: lustre-OST0000: new disk, initializing [ 179.297606] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 179.303368] Lustre: 8444:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 179.364830] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 184.586461] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 184.934383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 184.952343] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 185.023357] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 196.880753] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 197.025905] Lustre: lustre-OST0001: new disk, initializing [ 197.028646] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 197.036435] Lustre: 9515:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 197.092729] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 202.907324] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 204.323812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 204.337635] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 204.406989] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 213.659595] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 219.574181] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 224.829486] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing check_logdir /tmp/testlogs/ [ 229.092987] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing yml_node [ 232.989251] Lustre: DEBUG MARKER: Client: 2.17.54.84 [ 235.103383] Lustre: DEBUG MARKER: MDS: 2.17.54.84 [ 237.405232] Lustre: DEBUG MARKER: OSS: 2.17.54.84 [ 238.761693] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Aug 15 14:38:35 EDT 2026 [ 253.069985] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 254.572361] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 270.816704] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 270.818328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 270.839915] Lustre: Skipped 1 previous similar message [ 270.840237] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 270.880186] Lustre: Skipped 3 previous similar messages [ 275.937761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 275.944090] Lustre: Skipped 3 previous similar messages [ 281.063488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 281.075058] Lustre: Skipped 3 previous similar messages [ 284.128239] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 284.312324] Lustre: server umount lustre-MDT0000 complete [ 287.766624] LustreError: 6499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786819165 with bad export cookie 8909822128774638097 [ 287.774855] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 287.776814] LustreError: 6499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 291.295893] LustreError: 6513:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 291.296260] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 291.303462] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 291.312958] LustreError: 6513:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 291.331681] Lustre: Skipped 3 previous similar messages [ 292.323086] LustreError: 10188:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 292.343043] LustreError: 10188:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 297.439513] LustreError: 6514:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 297.439678] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 297.464321] LustreError: 6514:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 297.486663] Lustre: Skipped 4 previous similar messages [ 302.047524] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 302.276990] Lustre: server umount lustre-MDT0001 complete [ 306.032548] Lustre: server umount lustre-OST0000 complete [ 309.750867] Lustre: server umount lustre-OST0001 complete [ 315.435692] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 323.773280] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 352.171283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 352.319944] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 352.344973] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 352.423189] Lustre: lustre-MDT0000: new disk, initializing [ 352.491248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 352.502453] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 356.362248] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 365.557160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 365.684723] Lustre: 15067:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 365.748194] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 365.760067] Lustre: Skipped 1 previous similar message [ 365.847821] Lustre: lustre-MDT0001: new disk, initializing [ 365.964176] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 366.002366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 366.016833] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 369.927563] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 377.641429] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 382.275415] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 387.847600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 388.079262] Lustre: lustre-OST0000: new disk, initializing [ 388.081981] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 388.086638] Lustre: 17037:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 388.166606] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 389.412383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 389.417685] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 389.497414] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 393.309805] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 401.238798] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 405.176416] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 405.351838] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 408.949310] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 409.114375] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 415.206818] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 415.208793] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 415.209043] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 415.222417] Lustre: Skipped 1 previous similar message [ 416.829658] Lustre: server umount lustre-OST0000 complete [ 420.321510] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 420.346228] Lustre: Skipped 1 previous similar message [ 426.170119] Lustre: server umount lustre-MDT0000 complete [ 429.360413] LustreError: 15060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786819307 with bad export cookie 8909822128774639637 [ 429.361106] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 429.375687] LustreError: 15060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 429.646790] Lustre: server umount lustre-MDT0001 complete [ 434.596604] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 14:41:50 (1786819310) [ 454.027258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 454.683858] LustreError: 19910:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 454.769319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 459.388658] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 459.754275] LustreError: 19911:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 466.291972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 469.702829] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 475.311081] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 479.590697] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 481.695066] Lustre: Failing over lustre-MDT0000 [ 481.769518] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 481.777279] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 481.787204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 481.795505] Lustre: Skipped 5 previous similar messages [ 481.909187] Lustre: server umount lustre-MDT0000 complete [ 484.323085] LustreError: 21025:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 484.338791] LustreError: 21025:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 499.157666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 499.238459] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 502.582618] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 504.805943] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 504.810825] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 504.836093] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 510.060171] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 515.041275] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 515.048905] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 515.066453] Lustre: Skipped 1 previous similar message [ 515.070921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 519.979217] Lustre: server umount lustre-MDT0000 complete [ 522.213040] LustreError: 19910:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 522.230811] LustreError: 19910:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 523.165984] LustreError: 19890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786819400 with bad export cookie 8909822128774641051 [ 523.171361] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 523.182336] LustreError: 19890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 523.438596] Lustre: server umount lustre-MDT0001 complete [ 531.975328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 532.456122] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 532.467035] Lustre: Skipped 2 previous similar messages [ 536.679300] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 546.706587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 550.738982] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 557.028473] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 561.594308] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 563.639154] Lustre: Failing over lustre-MDT0000 [ 563.916340] Lustre: server umount lustre-MDT0000 complete [ 567.776695] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 567.778618] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 567.793344] Lustre: Skipped 1 previous similar message [ 567.800784] LustreError: 23973:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 567.822748] LustreError: 23973:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 582.518410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 582.621528] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 586.368708] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 588.263249] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 588.276920] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 588.301340] Lustre: Skipped 1 previous similar message [ 588.329572] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 594.604946] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 598.496090] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 598.501892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 598.509780] Lustre: Skipped 2 previous similar messages [ 602.596260] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 602.611510] Lustre: Skipped 1 previous similar message [ 604.671903] Lustre: server umount lustre-MDT0000 complete [ 608.409719] LustreError: 23227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786819486 with bad export cookie 8909822128774641989 [ 608.415754] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 608.424517] LustreError: 23227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 608.713873] Lustre: server umount lustre-MDT0001 complete [ 616.449272] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 14:44:52 (1786819492) [ 629.274954] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 640.090587] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 673.723654] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 684.492993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 684.706620] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 684.730187] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 684.796899] Lustre: lustre-MDT0000: new disk, initializing [ 684.867920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 684.875641] Lustre: Skipped 2 previous similar messages [ 684.891115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 689.044434] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 698.168950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 698.297969] Lustre: 29616:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 698.597427] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 703.042580] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 711.209950] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 716.693245] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 721.926068] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 722.173888] Lustre: 31584:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 724.021492] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 724.052614] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 724.113046] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 724.144838] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 727.420677] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 735.378274] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 741.823686] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 759.776900] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 759.791779] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 759.808846] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 759.812382] Lustre: Skipped 2 previous similar messages [ 761.753662] Lustre: server umount lustre-OST0000 complete [ 779.232963] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 779.483971] Lustre: server umount lustre-MDT0000 complete [ 780.771281] LustreError: 29627:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 780.800259] LustreError: 29627:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 783.227046] LustreError: 29609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786819661 with bad export cookie 8909822128774642395 [ 783.230383] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 783.237131] LustreError: 29609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 783.908475] Lustre: server umount lustre-MDT0001 complete [ 795.039846] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 798.160678] Key type lgssc unregistered [ 798.477532] LNet: 33560:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 798.485881] LNetError: 33560:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 798.498561] LNet: Removed LNI 192.168.206.157@tcp [ 799.425148] Key type .llcrypt unregistered [ 799.427633] Key type ._llcrypt unregistered [ 821.669493] Key type ._llcrypt registered [ 821.675115] Key type .llcrypt registered [ 821.788188] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 822.551294] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 822.559612] alg: No test for adler32 (adler32-zlib) [ 823.586145] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [ 823.782307] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 825.431372] Key type lgssc registered [ 826.380389] Lustre: Echo OBD driver; http://www.lustre.org/ [ 832.774254] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 832.798523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 834.184089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 837.963199] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 843.763079] LustreError: 34978:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 843.784695] LustreError: 34978:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 844.754641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 845.061329] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 849.322363] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 854.956425] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 860.052503] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 865.607537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 865.930608] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 870.083118] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 871.173406] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 876.857318] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 889.632636] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 14:49:25 (1786819765) [ 891.468975] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 893.242855] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 898.523487] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 14:49:34 (1786819774) [ 906.043874] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 14:49:42 (1786819782) [ 906.842238] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 913.785786] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 913.795521] Lustre: Skipped 2 previous similar messages [ 915.323898] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 14:49:51 (1786819791) [ 918.710430] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 918.715135] Lustre: Skipped 3 previous similar messages [ 934.760815] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 934.765352] Lustre: Skipped 6 previous similar messages [ 937.403436] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 14:50:13 (1786819813) [ 940.088679] Lustre: *** cfs_fail_loc=131b, val=2*** [ 942.971219] Lustre: *** cfs_fail_loc=131b, val=2*** [ 951.592508] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 14:50:27 (1786819827) [ 953.604408] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 953.610940] Lustre: Skipped 3 previous similar messages [ 965.245205] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 14:50:41 (1786819841) [ 977.888750] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 977.904831] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 977.919358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 980.960681] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 980.966870] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 980.985479] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 986.083145] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 986.093707] Lustre: Skipped 2 previous similar messages [ 991.202228] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 992.223250] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 992.415063] Lustre: server umount lustre-OST0000 complete [ 996.323436] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 996.334582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 996.340873] Lustre: Skipped 1 previous similar message [ 996.346937] Lustre: Skipped 1 previous similar message [ 1002.594233] Lustre: server umount lustre-MDT0000 complete [ 1006.562777] LustreError: 34973:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1006.583675] LustreError: 34973:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1007.076748] LustreError: 36820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786819884 with bad export cookie 18244033865961608179 [ 1007.093378] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1007.096652] LustreError: 36820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1007.481623] Lustre: server umount lustre-MDT0001 complete [ 1018.425386] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 1020.900881] Key type lgssc unregistered [ 1021.177167] LNet: 42822:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1021.188038] LNetError: 42822:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1021.205564] LNet: Removed LNI 192.168.206.157@tcp [ 1022.005972] Key type .llcrypt unregistered [ 1022.007979] Key type ._llcrypt unregistered [ 1039.049966] Key type ._llcrypt registered [ 1039.052594] Key type .llcrypt registered [ 1039.156920] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 1053.538758] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 1054.353825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1054.371221] alg: No test for adler32 (adler32-zlib) [ 1055.393888] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [ 1055.648233] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 1057.391491] Key type lgssc registered [ 1058.245680] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1089.421572] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1089.451032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1090.756571] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1090.817400] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1090.907860] Lustre: lustre-MDT0000: new disk, initializing [ 1091.001775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1091.016766] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1094.860742] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1103.473243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1103.650431] Lustre: 45974:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1103.712654] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1103.717235] Lustre: Skipped 1 previous similar message [ 1103.811754] Lustre: lustre-MDT0001: new disk, initializing [ 1103.934817] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1103.969888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1103.980565] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1108.296662] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1116.188280] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1120.658857] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1126.087093] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1126.303460] Lustre: lustre-OST0000: new disk, initializing [ 1126.309406] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1126.316952] Lustre: 47943:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1126.381097] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1131.441122] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1132.070589] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1132.079982] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1132.156506] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1139.662809] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1143.550567] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1143.743610] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1147.870268] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1148.071311] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1152.481887] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1152.488554] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1152.503043] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1153.002380] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1153.017418] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1153.034404] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1158.124687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1163.242979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1163.251098] Lustre: Skipped 1 previous similar message [ 1164.255496] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1164.463793] Lustre: server umount lustre-OST0000 complete [ 1168.353250] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1168.376715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1168.386751] Lustre: Skipped 2 previous similar messages [ 1178.594651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1178.604414] Lustre: Skipped 3 previous similar messages [ 1182.177478] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1182.342928] Lustre: server umount lustre-MDT0000 complete [ 1185.260244] LustreError: 47942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786820063 with bad export cookie 2389463271252349945 [ 1185.267415] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1185.269279] LustreError: 47942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1185.496650] Lustre: server umount lustre-MDT0001 complete [ 1193.651617] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 14:54:29 (1786820069) [ 1200.104115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1200.723964] LustreError: 50404:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1200.756047] LustreError: 50404:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1200.868966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1205.833959] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1206.242837] LustreError: 50405:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1210.337479] LustreError: 50404:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1214.261199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1214.607284] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1218.855551] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1225.507228] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1230.902918] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1237.003294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1237.358849] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1242.969260] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1249.304331] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1254.216358] LustreError: 52932:0:(llog.c:444:llog_init_handle()) llog type is not specified! [ 1255.176150] Lustre: Modifying parameter general.timeout=1 in log params [ 1256.184893] Lustre: Modifying parameter general.timeout=2 in log params [ 1259.014049] Lustre: Modifying parameter general.timeout=5 in log params [ 1259.021085] Lustre: Skipped 2 previous similar messages [ 1263.206130] Lustre: Modifying parameter general.timeout=9 in log params [ 1263.209441] Lustre: Skipped 3 previous similar messages [ 1272.110753] Lustre: Modifying parameter general.timeout=19 in log params [ 1272.116056] Lustre: Skipped 9 previous similar messages [ 1288.297486] Lustre: Modifying parameter general.timeout=38 in log params [ 1288.308053] Lustre: Skipped 18 previous similar messages [ 1320.819907] Lustre: Modifying parameter general.timeout=77 in log params [ 1320.823376] Lustre: Skipped 38 previous similar messages [ 1375.943695] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 14:57:32 (1786820252) [ 1378.439520] Lustre: 59873:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1380.040664] Lustre: 59972:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1383.151618] Lustre: 60169:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1383.162592] Lustre: 60169:0:(mgs_llog.c:1447:mgs_modify_param()) Skipped 1 previous similar message [ 1385.493919] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=5000 in log params [ 1385.500799] Lustre: Skipped 59 previous similar messages [ 1408.653196] Lustre: 61741:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1408.666785] Lustre: 61741:0:(mgs_llog.c:1447:mgs_modify_param()) Skipped 2 previous similar messages [ 1408.696258] LustreError: 61741:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1412.223396] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 14:58:08 (1786820288) [ 1424.353631] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1424.369672] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1424.396303] Lustre: Skipped 1 previous similar message [ 1424.422182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1430.639744] Lustre: server umount lustre-MDT0000 complete [ 1432.037082] LustreError: 50404:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1432.048778] LustreError: 50404:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1433.952699] LustreError: 52981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786820311 with bad export cookie 2389463271252350834 [ 1433.954556] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1433.960751] LustreError: 52981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1434.326935] Lustre: server umount lustre-MDT0001 complete [ 1448.054570] Lustre: server umount lustre-OST0000 complete [ 1470.021675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1470.241680] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1470.540184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1474.154846] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1480.909380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1480.998277] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1481.020586] Lustre: 64272:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1485.243328] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1491.284953] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1495.847233] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1501.764029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1501.880802] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1507.113744] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1513.945428] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1515.477093] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1515.484655] Lustre: Skipped 16 previous similar messages [ 1515.522785] Lustre: 66836:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1522.659308] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1522.665879] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1522.683326] Lustre: Skipped 2 previous similar messages [ 1522.687203] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1522.691078] Lustre: Skipped 3 previous similar messages [ 1526.578419] Lustre: server umount lustre-OST0000 complete [ 1532.386512] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1532.406239] LustreError: Skipped 1 previous similar message [ 1532.412674] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1532.435883] Lustre: Skipped 1 previous similar message [ 1544.159120] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1544.284260] Lustre: server umount lustre-MDT0000 complete [ 1546.832756] LustreError: 64263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786820424 with bad export cookie 2389463271252353872 [ 1546.838153] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1546.842979] LustreError: 64263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1547.074386] Lustre: server umount lustre-MDT0001 complete [ 1556.939780] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 1559.427950] Key type lgssc unregistered [ 1559.757736] LNet: 68045:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1559.763650] LNetError: 68045:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1559.785451] LNet: Removed LNI 192.168.206.157@tcp [ 1560.484564] Key type .llcrypt unregistered [ 1560.489104] Key type ._llcrypt unregistered [ 1580.175413] Key type ._llcrypt registered [ 1580.177377] Key type .llcrypt registered [ 1580.261967] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 1581.323972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1581.331693] alg: No test for adler32 (adler32-zlib) [ 1582.357131] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [ 1582.553479] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 1584.184363] Key type lgssc registered [ 1585.609696] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1592.343390] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1592.371761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1593.780262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1597.736540] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1604.064397] LustreError: 69449:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1604.087056] LustreError: 69449:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1605.366838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1605.928043] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1609.898874] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1616.449312] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1621.344345] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1626.948708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1627.352341] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1632.435628] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1638.414311] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1645.844280] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1652.175576] Lustre: server umount lustre-MDT0000 complete [ 1654.242920] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1654.247987] LustreError: 69450:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1654.263935] Lustre: Skipped 1 previous similar message [ 1654.949803] LustreError: 69430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786820532 with bad export cookie 9557284625856006484 [ 1654.952791] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1654.961362] LustreError: 69430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1657.215737] Lustre: server umount lustre-MDT0001 complete [ 1675.827636] Lustre: server umount lustre-OST0000 complete [ 1697.851849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1698.167525] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1698.220064] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1698.635108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1702.946965] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1709.967349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1710.115065] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1710.158295] Lustre: 74313:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1711.007968] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1715.716797] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1722.456312] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1727.759394] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1733.784240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1733.961872] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1734.403416] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1740.100698] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1747.000359] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1748.559108] Lustre: 76890:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1750.176963] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1757.152077] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1757.158869] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1757.171109] Lustre: Skipped 1 previous similar message [ 1757.180271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1757.664289] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1757.681086] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1760.200590] Lustre: server umount lustre-OST0000 complete [ 1766.880242] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1766.884254] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1766.902013] Lustre: Skipped 1 previous similar message [ 1766.910960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1769.817106] Lustre: server umount lustre-MDT0000 complete [ 1773.031429] LustreError: 74320:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1773.051801] LustreError: 74320:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1773.336615] LustreError: 74306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786820651 with bad export cookie 9557284625856007289 [ 1773.345411] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1773.345507] LustreError: 74306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1773.714475] Lustre: server umount lustre-MDT0001 complete [ 1783.189385] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 1785.647142] Key type lgssc unregistered [ 1785.911373] LNet: 78099:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1785.923690] LNetError: 78099:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1785.945680] LNet: Removed LNI 192.168.206.157@tcp [ 1786.775480] Key type .llcrypt unregistered [ 1786.777615] Key type ._llcrypt unregistered [ 1808.284522] Key type ._llcrypt registered [ 1808.286459] Key type .llcrypt registered [ 1808.403665] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 1809.344557] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1809.450966] alg: No test for adler32 (adler32-zlib) [ 1810.540611] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [ 1810.755371] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 1812.407176] Key type lgssc registered [ 1813.292234] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1819.280054] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1819.296091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1820.721249] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1824.446979] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1830.893548] LustreError: 79504:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1830.910348] LustreError: 79504:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1832.118103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1832.617465] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1836.323826] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1842.291117] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1847.416352] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1853.236854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1853.714357] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1858.253699] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1864.235414] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1873.946767] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1874.842067] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1876.259949] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1876.265560] Lustre: Skipped 1 previous similar message [ 1878.971725] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1878.976388] Lustre: Skipped 2 previous similar messages [ 1883.415593] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1883.420179] Lustre: Skipped 4 previous similar messages [ 1891.968163] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1891.976540] Lustre: Skipped 10 previous similar messages [ 1908.331691] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1908.339141] Lustre: Skipped 22 previous similar messages [ 1940.810644] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1940.822248] Lustre: Skipped 44 previous similar messages [ 1954.361365] Lustre: 87591:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1956.420299] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 15:07:12 (1786820832) [ 1957.891914] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1959.495895] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 15:07:15 (1786820835) [ 1963.488066] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1963.493749] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1963.504405] Lustre: Skipped 1 previous similar message [ 1963.506035] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1963.513112] Lustre: Skipped 1 previous similar message [ 1968.610089] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1968.615226] Lustre: Skipped 1 previous similar message [ 1973.729877] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1973.745801] Lustre: Skipped 1 previous similar message [ 1977.315618] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1977.438671] Lustre: server umount lustre-OST0000 complete [ 1980.900965] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1980.911338] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1980.936267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1983.976754] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1986.888903] Lustre: server umount lustre-MDT0000 complete [ 1989.087975] LustreError: 79504:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1989.097656] LustreError: 79504:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1989.910561] LustreError: 81354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786820867 with bad export cookie 2215449446155169649 [ 1989.912174] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1989.928782] LustreError: 81354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1990.176766] Lustre: server umount lustre-MDT0001 complete [ 1998.861702] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 2001.121983] Key type lgssc unregistered [ 2001.364798] LNet: 88842:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2001.371963] LNetError: 88842:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2001.388489] LNet: Removed LNI 192.168.206.157@tcp [ 2002.082702] Key type .llcrypt unregistered [ 2002.086313] Key type ._llcrypt unregistered [ 2005.412757] Key type ._llcrypt registered [ 2005.418239] Key type .llcrypt registered [ 2005.504260] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2009.582697] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules [ 2010.353524] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2010.387163] alg: No test for adler32 (adler32-zlib) [ 2011.401852] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [ 2011.638560] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 2013.271160] Key type lgssc registered [ 2014.394958] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2015.442879] LustreError: 90044:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2031.796962] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 2037.198228] LustreError: 90044:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2040.153191] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2040.168969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2041.588241] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2045.018925] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2051.781947] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 15:08:48 (1786820928) [ 2052.064413] LustreError: 91069:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2052.072545] LustreError: 91069:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2055.319418] LustreError: 91101:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2055.340211] LustreError: 91101:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2055.350737] LustreError: 91101:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 14, retries 0, failed: rc = -5 [ 2055.717481] Lustre: server umount lustre-MDT0000 complete [ 2065.606591] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 2067.736425] Key type lgssc unregistered [ 2067.941306] LNet: 92593:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2067.946088] LNetError: 92593:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2067.966923] LNet: Removed LNI 192.168.206.157@tcp [ 2068.607654] Key type .llcrypt unregistered [ 2068.610248] Key type ._llcrypt unregistered [ 2083.361467] Key type ._llcrypt registered [ 2083.369636] Key type .llcrypt registered [ 2083.471601] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 2084.425068] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2084.566984] alg: No test for adler32 (adler32-zlib) [ 2085.731088] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [ 2085.921574] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 2087.591205] Key type lgssc registered [ 2088.931191] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2095.216221] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2095.248893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2096.736035] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2100.688796] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2106.852909] LustreError: 93791:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2106.876326] LustreError: 93791:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2108.277833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2108.858715] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2112.881391] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2119.918646] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2124.031853] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2128.543545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2128.976942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2133.447367] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2139.162853] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2183.538263] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 15:10:59 (1786821059) [ 2185.053158] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2186.495683] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 15:11:02 (1786821062) [ 2190.818343] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2190.827157] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2190.844429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2192.867352] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2192.877085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2192.886649] Lustre: Skipped 1 previous similar message [ 2192.888932] Lustre: Skipped 1 previous similar message [ 2196.043576] Lustre: server umount lustre-MDT0000 complete [ 2197.985643] LustreError: 93790:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2198.000733] LustreError: 93790:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2199.541202] LustreError: 94530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786821077 with bad export cookie 7968702970720885362 [ 2199.553336] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2199.568535] LustreError: 94530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2200.056383] Lustre: server umount lustre-MDT0001 complete [ 2214.702527] Lustre: server umount lustre-OST0000 complete [ 2225.174971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2225.646175] LustreError: 98062:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2225.824285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2230.041762] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2231.268183] LustreError: 98063:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2235.363377] LustreError: 98062:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2237.295093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2237.958586] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2242.118773] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2248.983420] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2253.897972] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2264.157459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2264.400699] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2264.409225] LustreError: 98049:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2264.431705] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2264.446383] LustreError: 100145:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2264.456412] LustreError: 100145:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2264.463319] LustreError: 100145:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2264.538403] Lustre: server umount lustre-OST0000 complete [ 2264.542173] LustreError: 100145:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2271.629511] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2271.876097] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2271.889637] LustreError: 98052:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2271.900453] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2271.913835] LustreError: 100606:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2271.922652] LustreError: 100606:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2271.928580] LustreError: 100606:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2272.007421] Lustre: server umount lustre-OST0000 complete [ 2272.012019] LustreError: 100606:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2279.690103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2279.822442] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2279.830510] Lustre: Found index 0 for lustre-OST0000, updating log [ 2279.834570] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2279.902512] Lustre: lustre-OST0000: new disk, initializing [ 2279.905577] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2279.911986] Lustre: 101140:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2280.185858] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2284.942242] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2287.669539] Lustre: Failing over lustre-OST0000 [ 2287.769110] Lustre: server umount lustre-OST0000 complete [ 2292.705420] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2292.709437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2292.716233] Lustre: Skipped 1 previous similar message [ 2292.721290] Lustre: Skipped 1 previous similar message [ 2297.300381] Lustre: server umount lustre-MDT0000 complete [ 2297.825740] LustreError: 98062:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2297.842498] LustreError: 98062:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2300.256414] LustreError: 98042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786821178 with bad export cookie 7968702970720888106 [ 2300.260097] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2300.265290] LustreError: 98042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2310.769587] Lustre: server umount lustre-MDT0001 complete [ 2317.163698] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 15:13:13 (1786821193) [ 2332.008571] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 2342.124594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2342.423390] LustreError: 103487:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2342.541112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2346.618706] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2354.540361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2360.036284] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2362.570423] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2368.251312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2368.881053] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2373.948128] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2380.140150] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2380.162045] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2380.164261] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2381.163461] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2381.264135] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2381.368592] Lustre: lustre-OST0001: new disk, initializing [ 2381.372621] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2381.377956] Lustre: 105804:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2381.667269] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2381.676170] Lustre: Skipped 2 previous similar messages [ 2386.804649] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2388.023393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2388.036438] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2388.112764] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2393.704456] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2399.352814] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2403.811455] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2403.816846] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2403.819304] Lustre: Skipped 2 previous similar messages [ 2403.828331] Lustre: Skipped 2 previous similar messages [ 2408.929945] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2408.945598] Lustre: Skipped 3 previous similar messages [ 2409.323811] Lustre: server umount lustre-MDT0001 complete [ 2409.604606] LustreError: 104958:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2409.624764] LustreError: 104958:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2419.485453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2419.570755] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2419.576305] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2419.684404] Lustre: 103477:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2421.764796] LustreError: 107381:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2421.778677] LustreError: 107381:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2421.804394] LustreError: 103480:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2421.981949] LustreError: 107388:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.206.157@tcp: cfg command failed: rc = -17 [ 2422.005603] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2422.005603] [ 2425.524813] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2425.864614] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2434.564844] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 15:15:10 (1786821310) [ 2448.552421] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 2463.495460] Lustre: 109417:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2483.118450] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2486.263095] Lustre: 110551:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2563.552068] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2563.558799] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2563.568851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2567.742878] Lustre: server umount lustre-MDT0000 complete [ 2569.185838] LustreError: 103483:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2569.206108] LustreError: 103483:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 2570.462481] LustreError: 105803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786821448 with bad export cookie 7968702970720889030 [ 2570.463837] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2570.476393] LustreError: 105803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2570.677727] Lustre: server umount lustre-MDT0001 complete [ 2583.590525] Lustre: server umount lustre-OST0000 complete [ 2596.708471] Lustre: server umount lustre-OST0001 complete [ 2601.230730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2609.777375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2617.668257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2626.782678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2643.807406] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 2650.947563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2650.980331] Lustre: lustre-MDT0000: reset Object Index mappings [ 2651.362324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2651.367816] Lustre: Skipped 1 previous similar message [ 2654.264316] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2661.134823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2661.148878] Lustre: lustre-MDT0001: reset Object Index mappings [ 2665.264532] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2667.734817] Lustre: 115648:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2674.008573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2677.689265] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:354 to 0x280000401:386) [ 2679.698251] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2686.828416] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2686.965063] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:226 to 0x280000400:257) [ 2692.599482] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 2692.629846] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:195 to 0x2c0000400:225) [ 2693.200864] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2700.018220] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2703.304381] Lustre: 117524:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2746.671755] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 15:20:22 (1786821622) [ 2773.984320] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2773.996287] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2774.013568] Lustre: Skipped 3 previous similar messages [ 2774.017690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2774.020377] Lustre: Skipped 4 previous similar messages [ 2779.623828] LustreError: 114494:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2779.645713] LustreError: 114494:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2781.513817] Lustre: server umount lustre-MDT0000 complete [ 2784.254962] LustreError: 114481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786821662 with bad export cookie 7968702970720935909 [ 2784.262831] LustreError: 114481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2784.264322] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2790.767931] Lustre: server umount lustre-MDT0001 complete [ 2808.287146] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2808.410583] Lustre: server umount lustre-OST0000 complete [ 2818.333345] Lustre: server umount lustre-OST0001 complete [ 2825.041760] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 2835.394839] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 2869.321213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2869.649801] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2869.692475] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2869.863726] Lustre: lustre-MDT0000: new disk, initializing [ 2869.993669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2870.007415] Lustre: Skipped 3 previous similar messages [ 2870.030574] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2876.312838] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2882.759988] Lustre: server umount lustre-MDT0000 complete [ 2891.514505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2891.784072] Lustre: Found index 0 for lustre-MDT0000, updating log [ 2891.825147] Lustre: 121790:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2891.850101] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 2895.977690] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2900.188657] Lustre: server umount lustre-MDT0000 complete [ 2908.057226] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 15:23:04 (1786821784) [ 2909.506106] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 2910.937957] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 15:23:07 (1786821787) [ 2925.077426] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 2934.378453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2939.004300] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2946.104910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2946.268340] Lustre: 123838:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2946.322535] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2946.326404] Lustre: Skipped 1 previous similar message [ 2946.401627] Lustre: lustre-MDT0001: new disk, initializing [ 2946.557834] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2946.569718] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2951.075700] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2955.705335] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2961.756291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2961.985588] Lustre: lustre-OST0000: new disk, initializing [ 2961.993226] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2962.002964] Lustre: 125416:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2963.088185] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2963.103307] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2963.222398] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2968.526834] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2977.930534] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2978.098563] Lustre: lustre-OST0001: new disk, initializing [ 2978.104175] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2978.110378] Lustre: 126286:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2979.720376] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2979.745460] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2979.845737] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2983.861668] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2991.774140] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2994.838992] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3322.773017] Lustre: 132690:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3322.780202] Lustre: 132690:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3322.785270] Lustre: 132690:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3322.789873] Lustre: 132690:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3322.797547] Lustre: 132690:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3322.804674] Lustre: 132690:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3556.690495] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 15:33:52 (1786822432) [ 3571.168508] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3571.175386] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3571.184030] Lustre: Skipped 5 previous similar messages [ 3571.187834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3571.191586] Lustre: Skipped 8 previous similar messages [ 3573.730322] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3573.749186] Lustre: Skipped 2 previous similar messages [ 3577.453554] Lustre: server umount lustre-MDT0000 complete [ 3578.860830] LustreError: 125410:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3578.879620] LustreError: 125410:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 3580.463914] LustreError: 127088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786822458 with bad export cookie 7968702970721001730 [ 3580.472415] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3580.476936] LustreError: 127088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3583.971228] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3583.982943] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3583.988258] Lustre: Skipped 1 previous similar message [ 3584.003876] Lustre: Skipped 5 previous similar messages [ 3586.920105] Lustre: server umount lustre-MDT0001 complete [ 3596.473050] Lustre: server umount lustre-OST0000 complete [ 3605.812792] Lustre: server umount lustre-OST0001 complete [ 3611.402617] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 3620.061509] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 3648.119117] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3648.303381] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3648.319136] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3648.382749] Lustre: lustre-MDT0000: new disk, initializing [ 3648.453641] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3648.459132] Lustre: Skipped 5 previous similar messages [ 3648.479751] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3651.842238] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 3660.519947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3660.687348] Lustre: 135729:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3660.743648] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3660.747122] Lustre: Skipped 1 previous similar message [ 3660.809964] Lustre: lustre-MDT0001: new disk, initializing [ 3660.910442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3660.917916] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3665.439608] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 3673.862210] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3679.646572] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3685.760226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3685.973675] Lustre: lustre-OST0000: new disk, initializing [ 3685.976220] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3685.980083] Lustre: 137699:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3687.846304] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3687.854278] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3687.943219] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3690.444835] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 3697.404349] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3704.353951] Lustre: lustre-MDD0000: changelog on [ 3706.593289] Lustre: lustre-MDD0001: changelog on [ 5262.472974] Lustre: lustre-MDD0001: changelog off [ 5264.169687] Lustre: 139161:0:(llog_cat.c:977:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 5265.005894] Lustre: lustre-MDD0000: changelog off [ 5275.105901] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5275.111463] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5275.119473] LustreError: Skipped 1 previous similar message [ 5275.121965] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5275.137193] Lustre: Skipped 1 previous similar message [ 5275.149189] Lustre: Skipped 3 previous similar messages [ 5276.821582] Lustre: server umount lustre-OST0000 complete [ 5283.808972] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5283.820383] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5283.847608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5287.298625] Lustre: server umount lustre-MDT0000 complete [ 5290.465774] LustreError: 139161:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5290.473711] LustreError: 139161:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5290.988231] LustreError: 135722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786824168 with bad export cookie 7968702970721822228 [ 5290.994568] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5291.011254] LustreError: 135722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5291.273611] Lustre: server umount lustre-MDT0001 complete [ 5301.961075] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 5304.550527] Key type lgssc unregistered [ 5304.804457] LNet: 146802:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5304.818785] LNetError: 146802:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5304.855961] LNet: Removed LNI 192.168.206.157@tcp [ 5305.563142] Key type .llcrypt unregistered [ 5305.564918] Key type ._llcrypt unregistered [ 5325.021321] Key type ._llcrypt registered [ 5325.025334] Key type .llcrypt registered [ 5325.130330] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 5343.344494] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 5345.083766] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5345.508195] alg: No test for adler32 (adler32-zlib) [ 5346.961841] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [ 5347.413697] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 5349.322454] Key type lgssc registered [ 5351.132350] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5384.516586] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5384.553896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5385.794197] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5385.845403] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5385.960610] Lustre: lustre-MDT0000: new disk, initializing [ 5386.026587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5386.041773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5390.115634] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 5400.488523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5400.606990] Lustre: 150000:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5400.649189] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5400.651827] Lustre: Skipped 1 previous similar message [ 5400.764840] Lustre: lustre-MDT0001: new disk, initializing [ 5400.841131] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5400.875095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5400.889672] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5406.234280] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 5415.598879] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5421.334226] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5428.241561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5428.527994] Lustre: lustre-OST0000: new disk, initializing [ 5428.531262] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5428.536385] Lustre: 151969:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5428.614359] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5435.119237] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 5435.445451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5435.451783] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5435.559887] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5444.144956] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5449.245551] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_module obdecho/obdecho [ 5451.009446] LustreError: 152971:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 5451.014604] LustreError: 152971:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 5452.822960] LustreError: 153068:0:(echo_client.c:1883:echo_md_handler()) Can not resolve the path : rc = -2 [ 5466.594165] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5466.611036] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5466.616033] Lustre: Skipped 2 previous similar messages [ 5466.629799] Lustre: Skipped 2 previous similar messages [ 5470.909918] Lustre: server umount lustre-MDT0000 complete [ 5474.405392] LustreError: 149994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786824352 with bad export cookie 14240205320507659299 [ 5474.419372] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5474.686930] Lustre: server umount lustre-MDT0001 complete [ 5487.595937] Lustre: server umount lustre-OST0000 complete [ 5508.373211] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 5518.201917] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 5550.903375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5551.158948] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5551.208269] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5551.294234] Lustre: lustre-MDT0000: new disk, initializing [ 5551.404552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5551.422294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5555.521982] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 5565.274235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5565.417421] Lustre: 156783:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5565.487642] LustreError: 156795:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5565.499159] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5565.500422] LustreError: 156795:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5565.512070] Lustre: Skipped 1 previous similar message [ 5565.593487] Lustre: lustre-MDT0001: new disk, initializing [ 5565.718662] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5565.769765] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5565.785731] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5569.885295] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 5579.802204] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5587.310490] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5595.645084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5595.931558] Lustre: lustre-OST0000: new disk, initializing [ 5595.935474] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5595.940552] Lustre: 158753:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5596.048717] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5602.141708] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 5602.409741] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5602.425144] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5602.546576] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5611.861351] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5615.945852] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5616.144631] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5620.690230] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5620.912810] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5623.267663] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5623.276394] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5623.287812] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5623.291204] Lustre: Skipped 1 previous similar message [ 5628.388615] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5628.394133] Lustre: Skipped 1 previous similar message [ 5628.856788] Lustre: server umount lustre-OST0000 complete [ 5633.505096] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5633.520530] Lustre: Skipped 1 previous similar message [ 5633.526753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5633.540126] Lustre: Skipped 1 previous similar message [ 5638.629911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5638.635941] Lustre: Skipped 1 previous similar message [ 5638.927245] Lustre: server umount lustre-MDT0000 complete [ 5642.231982] LustreError: 157579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786824520 with bad export cookie 14240205320507660258 [ 5642.234084] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5642.238059] LustreError: 157579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5642.525729] Lustre: server umount lustre-MDT0001 complete [ 5645.923888] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 16:08:42 (1786824522) [ 5647.712829] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 5649.149624] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 16:08:45 (1786824525) [ 5655.320539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5655.716034] LustreError: 161165:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5655.787986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5659.208571] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 5661.162312] LustreError: 161166:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5665.249372] LustreError: 161165:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5666.833025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5671.692933] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 5678.827557] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5684.299984] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5690.322380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5690.594565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5690.602824] Lustre: Skipped 1 previous similar message [ 5696.428414] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 5703.926516] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5709.978053] Lustre: server umount lustre-MDT0000 complete [ 5711.338626] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5711.354630] Lustre: Skipped 1 previous similar message [ 5711.358866] LustreError: 163561:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5711.379370] LustreError: 163561:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5714.221434] LustreError: 161145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786824592 with bad export cookie 14240205320507661140 [ 5714.225349] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5714.244045] LustreError: 161145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5714.614713] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.57@tcp (stopping) [ 5716.448425] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5719.457557] LustreError: 161161:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5719.477577] LustreError: 161161:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5720.614153] Lustre: server umount lustre-MDT0001 complete [ 5726.891073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5739.411907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5739.588572] LustreError: 165005:0:(llog.c:1646:llog_backup()) MGC192.168.206.157@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 5739.597612] Lustre: 165005:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.157@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 5745.126606] LustreError: 165010:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5745.150274] LustreError: 165010:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5779.942470] LustreError: 165029:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5779.956373] LustreError: 165029:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 29 previous similar messages [ 5800.416098] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 5800.422400] Lustre: 165005:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.206.157@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 5841.411662] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5845.581020] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 5846.496385] LustreError: 165012:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5846.502935] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5846.506920] LustreError: 165012:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 44 previous similar messages [ 5853.856903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5860.848527] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xc59f5f4a897cb48a [ 5860.866368] Lustre: MGC192.168.206.157@tcp: Connection restored to 0@lo (at 0@lo) [ 5861.014837] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.57@tcp (not set up) [ 5861.022927] Lustre: Skipped 3 previous similar messages [ 5865.439221] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 5866.473176] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5872.018267] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5873.340540] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5878.161547] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5879.976478] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5901.085793] Lustre: server umount lustre-MDT0000 complete [ 5902.304970] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5902.311064] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5902.314428] LustreError: Skipped 1 previous similar message [ 5902.340156] Lustre: Skipped 2 previous similar messages [ 5904.329606] LustreError: 161147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786824782 with bad export cookie 14240205320507667594 [ 5904.331131] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5904.350871] LustreError: 161147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5904.621926] Lustre: server umount lustre-MDT0001 complete [ 5918.501831] Lustre: server umount lustre-OST0000 complete [ 5935.971468] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 5945.896855] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 5977.638226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5977.914536] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5977.943265] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5978.031454] Lustre: lustre-MDT0000: new disk, initializing [ 5978.103555] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5978.126962] Lustre: Skipped 1 previous similar message [ 5978.173583] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5982.385893] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 5991.728600] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5991.825730] Lustre: 170111:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5991.992218] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5995.951253] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6004.401268] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6010.755645] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6017.734658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6017.963689] Lustre: lustre-OST0000: new disk, initializing [ 6017.966826] Lustre: Skipped 1 previous similar message [ 6017.970565] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6017.977242] Lustre: Skipped 2 previous similar messages [ 6017.984256] Lustre: 172078:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6024.319638] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6028.335776] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6028.342182] Lustre: Skipped 1 previous similar message [ 6028.346543] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6028.439840] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6032.760617] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6037.306641] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6037.506483] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6041.923744] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6042.215552] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6044.129828] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6044.129976] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6044.131692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6044.145503] Lustre: Skipped 1 previous similar message [ 6050.311793] Lustre: server umount lustre-OST0000 complete [ 6060.813831] Lustre: server umount lustre-MDT0000 complete [ 6064.608097] LustreError: 170123:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6064.627834] LustreError: 170123:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 16 previous similar messages [ 6064.994360] LustreError: 170914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786824942 with bad export cookie 14240205320507684429 [ 6064.997369] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6065.006491] LustreError: 170914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6065.618797] Lustre: server umount lustre-MDT0001 complete [ 6074.316350] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 16:15:50 (1786824950) [ 6080.158812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6084.442784] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6091.807730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6095.737105] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6102.675486] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6107.912500] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6114.412514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6121.150672] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6127.481970] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6136.858585] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 16:16:52 (1786825012) [ 6141.412502] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6141.414738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6141.422218] Lustre: Skipped 3 previous similar messages [ 6141.427212] Lustre: Skipped 7 previous similar messages [ 6141.428157] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6147.002513] Lustre: server umount lustre-OST0000 complete [ 6156.738969] Lustre: server umount lustre-MDT0000 complete [ 6160.462686] LustreError: 177462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786825038 with bad export cookie 14240205320507685311 [ 6160.471232] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6160.472524] LustreError: 177462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6160.793648] Lustre: server umount lustre-MDT0001 complete [ 6171.262803] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 6173.968977] Key type lgssc unregistered [ 6174.342665] LNet: 178213:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6174.359857] LNetError: 178213:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6174.379454] LNet: Removed LNI 192.168.206.157@tcp [ 6175.201897] Key type .llcrypt unregistered [ 6175.207881] Key type ._llcrypt unregistered [ 6191.361484] Key type ._llcrypt registered [ 6191.362994] Key type .llcrypt registered [ 6191.452228] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 6192.273475] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6192.425218] alg: No test for adler32 (adler32-zlib) [ 6193.719813] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [ 6194.026702] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 6195.687163] Key type lgssc registered [ 6197.047888] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6204.327084] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6204.341676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6219.871485] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6224.992513] LustreError: 179459:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.157@tcp: failed processing log, type 4: rc = -110 [ 6230.111443] LustreError: 179459:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6230.122312] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 6230.127666] LustreError: MGC192.168.206.157@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 6230.140749] LustreError: 179459:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 6230.151675] LustreError: 179459:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 6230.161123] LustreError: 179459:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 6230.229917] Lustre: server umount lustre-OST0000 complete [ 6230.234468] LustreError: 179459:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6237.667716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6239.465796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6244.322391] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6249.959906] LustreError: 179958:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6249.988977] LustreError: 179958:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6254.229654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6254.778868] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6259.163173] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6265.884479] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6270.587213] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6276.362390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6276.492320] LustreError: 181774:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6276.499988] Lustre: 181774:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.206.157@tcp: invalid local config log lustre-OST0000: rc = -22 [ 6276.614929] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6281.431339] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6287.709447] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6290.413963] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6290.419062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6290.428807] Lustre: Skipped 1 previous similar message [ 6290.436571] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6290.440534] Lustre: Skipped 1 previous similar message [ 6295.526591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6295.532294] Lustre: Skipped 3 previous similar messages [ 6296.528992] Lustre: server umount lustre-MDT0000 complete [ 6300.007123] LustreError: 179938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786825177 with bad export cookie 9468660596815244329 [ 6300.015764] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6300.028576] LustreError: 179938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6300.401883] Lustre: server umount lustre-MDT0001 complete [ 6315.051664] Lustre: server umount lustre-OST0000 complete [ 6323.129148] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 6331.928840] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 6369.384869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6369.809894] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6369.847395] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6370.006693] Lustre: lustre-MDT0000: new disk, initializing [ 6370.116900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6370.131818] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6374.418307] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6384.553705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6384.686773] Lustre: 185297:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6384.736271] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6384.739951] Lustre: Skipped 1 previous similar message [ 6384.853927] Lustre: lustre-MDT0001: new disk, initializing [ 6384.932149] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6384.973725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6384.992932] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6389.500596] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6397.761880] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6403.185186] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6408.867436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6409.055825] Lustre: lustre-OST0000: new disk, initializing [ 6409.059123] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6409.065806] Lustre: 187266:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6409.142193] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6414.238966] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6417.975402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6417.989973] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6418.052784] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6421.621303] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6425.855414] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6426.032119] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6429.477158] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6429.660168] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6433.263260] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6433.281660] Lustre: Skipped 2 previous similar messages [ 6433.294649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6437.257508] Lustre: server umount lustre-OST0000 complete [ 6441.440697] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6441.448930] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6441.465430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6441.472913] Lustre: Skipped 1 previous similar message [ 6443.488657] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6447.412466] Lustre: server umount lustre-MDT0000 complete [ 6448.608145] LustreError: 185305:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6448.615954] LustreError: 185305:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6450.875305] LustreError: 185289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786825328 with bad export cookie 9468660596815245050 [ 6450.879412] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6450.892386] LustreError: 185289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6451.222825] Lustre: server umount lustre-MDT0001 complete [ 6459.965940] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 16:22:15 (1786825335) [ 6475.719428] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 6478.201758] Key type lgssc unregistered [ 6478.479556] LNet: 190209:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6478.487100] LNetError: 190209:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6478.503809] LNet: Removed LNI 192.168.206.157@tcp [ 6479.224313] Key type .llcrypt unregistered [ 6479.227313] Key type ._llcrypt unregistered [ 6495.340661] Key type ._llcrypt registered [ 6495.343261] Key type .llcrypt registered [ 6495.424270] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 6496.581445] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6496.756459] alg: No test for adler32 (adler32-zlib) [ 6498.060923] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [ 6498.281364] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 6499.904196] Key type lgssc registered [ 6501.217573] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6508.171096] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6508.185592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6509.659595] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6513.592545] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6519.778484] LustreError: 191439:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6519.798820] LustreError: 191439:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6523.907801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6524.436826] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6528.745545] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6537.209155] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6543.202110] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6550.224940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6550.349910] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6550.351707] LustreError: 193305:0:(llog.c:1687:llog_backup()) MGC192.168.206.157@tcp: failed to backup log lustre-OST0000: rc = -28 [ 6550.355343] Lustre: 193305:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.206.157@tcp: can't backup local config lustre-OST0000: rc = -28 [ 6550.419635] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6555.994223] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6562.281874] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6564.832461] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6564.849354] Lustre: Skipped 1 previous similar message [ 6564.850951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6569.953159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6569.957721] Lustre: Skipped 3 previous similar messages [ 6570.321567] Lustre: server umount lustre-MDT0000 complete [ 6570.979355] LustreError: 193863:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6571.000039] LustreError: 193863:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6573.356972] LustreError: 191421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786825451 with bad export cookie 3845493803047219259 [ 6573.363432] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6573.366400] LustreError: 191421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6573.674957] Lustre: server umount lustre-MDT0001 complete [ 6586.431571] Lustre: server umount lustre-OST0000 complete [ 6594.472580] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 6605.428522] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 6638.725902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6638.937533] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6638.959690] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6639.027345] Lustre: lustre-MDT0000: new disk, initializing [ 6639.150550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6639.165443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6643.031608] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6652.573205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6652.728293] Lustre: 196832:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6652.774263] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6652.776670] Lustre: Skipped 1 previous similar message [ 6652.779328] LustreError: 197614:0:(llog.c:1687:llog_backup()) MGC192.168.206.157@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 6652.785628] LustreError: 197614:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 6652.790942] Lustre: 197614:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.157@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 6652.807530] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6652.814351] Lustre: Skipped 1 previous similar message [ 6652.903589] Lustre: lustre-MDT0001: new disk, initializing [ 6653.000780] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6653.030271] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6653.044816] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6656.999548] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6665.385166] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6670.097052] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6675.637540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6675.936741] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6675.940409] Lustre: Skipped 1 previous similar message [ 6675.943483] LustreError: 198776:0:(llog.c:1687:llog_backup()) MGC192.168.206.157@tcp: failed to backup log lustre-OST0000: rc = -28 [ 6675.951043] LustreError: 198776:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 6675.955555] Lustre: 198776:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.157@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 6675.963509] Lustre: 198776:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 6675.980370] Lustre: lustre-OST0000: new disk, initializing [ 6675.984613] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6675.992454] Lustre: 198800:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6676.099381] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6681.672913] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6682.170255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6682.187465] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6682.282899] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6690.196463] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6693.742386] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6693.935802] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6698.000546] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6698.388522] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6703.074482] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6703.081165] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6703.090557] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6703.094238] Lustre: Skipped 2 previous similar messages [ 6703.112563] Lustre: Skipped 1 previous similar message [ 6706.699464] Lustre: server umount lustre-OST0000 complete [ 6713.321068] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6713.334402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6713.344471] Lustre: Skipped 1 previous similar message [ 6717.594156] Lustre: server umount lustre-MDT0000 complete [ 6718.442430] LustreError: 196840:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6718.464203] LustreError: 196840:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6721.125849] LustreError: 198799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786825598 with bad export cookie 3845493803047219980 [ 6721.144117] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6721.145724] LustreError: 198799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6721.730538] Lustre: server umount lustre-MDT0001 complete [ 6730.008898] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 16:26:46 (1786825606) [ 6744.825655] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 6753.190883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6753.601835] LustreError: 201924:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6753.683387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6757.668748] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6758.883575] LustreError: 201925:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6765.411383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6769.349697] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6772.070298] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6778.538535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6778.865417] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 6778.877638] Lustre: Skipped 1 previous similar message [ 6784.937624] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6792.870662] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6793.091667] Lustre: lustre-OST0001: new disk, initializing [ 6793.096476] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6793.103517] Lustre: 204231:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6793.203470] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6793.209232] Lustre: Skipped 2 previous similar messages [ 6797.864592] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6802.984073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6802.989498] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6803.059416] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6807.643575] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6815.904436] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6817.718705] Lustre: DEBUG MARKER: ADD OST3 [ 6823.104092] loop: module loaded [ 6823.221759] loop0: detected capacity change from 0 to 204800000 [ 6823.606369] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6823.649782] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6823.693364] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6831.400952] Lustre: DEBUG MARKER: STOP OST3 [ 6831.612047] loop0: detected capacity change from 0 to 204800000 [ 6837.979992] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6838.136336] Lustre: lustre-OST0003: new disk, initializing [ 6838.152945] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 6838.161435] Lustre: 206285:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6839.612148] Lustre: *** cfs_fail_loc=2109, val=2*** [ 6839.705459] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 6839.711415] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 6839.763260] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 6841.660239] LustreError: 206414:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 6841.797346] Lustre: Failing over lustre-OST0003 [ 6841.985426] Lustre: server umount lustre-OST0003 complete [ 6844.932808] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6844.933823] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6844.942794] LustreError: 203412:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6844.942804] LustreError: 203412:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6844.966777] LustreError: Skipped 1 previous similar message [ 6846.573506] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6854.550270] Lustre: DEBUG MARKER: START OST3 again [ 6860.475432] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6860.665901] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 6860.669705] Lustre: Skipped 1 previous similar message [ 6860.683485] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 6862.239399] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 6862.335512] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 6862.336594] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6866.115944] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6886.022910] Lustre: Failing over lustre-OST0003 [ 6886.200344] Lustre: server umount lustre-OST0003 complete [ 6886.387076] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6886.415295] LustreError: 204239:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6886.419156] Lustre: Skipped 2 previous similar messages [ 6886.451757] LustreError: 204239:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 6903.777186] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6903.789609] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6903.809924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6909.648739] Lustre: server umount lustre-MDT0000 complete [ 6912.314894] LustreError: 201904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786825790 with bad export cookie 3845493803047220862 [ 6912.318714] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6912.328764] LustreError: 201904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6918.768710] Lustre: server umount lustre-MDT0001 complete [ 6922.018893] Lustre: server umount lustre-OST0000 complete [ 6925.402876] Lustre: server umount lustre-OST0001 complete [ 6931.853716] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 6939.167995] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 6976.014116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6976.312940] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6976.349471] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6976.446648] Lustre: lustre-MDT0000: new disk, initializing [ 6976.565398] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6980.612789] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6990.009041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6990.127392] Lustre: 211694:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6990.483441] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6990.490860] Lustre: Skipped 1 previous similar message [ 6990.548706] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6994.928858] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7005.453202] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7011.735478] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7018.501343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7018.774294] Lustre: lustre-OST0000: new disk, initializing [ 7018.777917] Lustre: Skipped 1 previous similar message [ 7018.784511] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7018.788825] Lustre: Skipped 2 previous similar messages [ 7018.794490] Lustre: 213661:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7020.654316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7020.662602] Lustre: Skipped 1 previous similar message [ 7020.664509] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7020.737906] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7024.603518] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7032.914236] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7037.231746] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7037.399735] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7042.113793] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7042.338446] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7046.629628] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7046.630619] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7046.638422] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7046.638510] Lustre: Skipped 6 previous similar messages [ 7046.650936] LustreError: Skipped 1 previous similar message [ 7046.698882] Lustre: Skipped 6 previous similar messages [ 7058.399145] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7058.517693] Lustre: server umount lustre-OST0000 complete [ 7062.497804] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7076.319537] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7076.504342] Lustre: server umount lustre-MDT0000 complete [ 7077.345197] LustreError: 211704:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7077.361423] LustreError: 211704:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 15 previous similar messages [ 7080.098389] LustreError: 211684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786825957 with bad export cookie 3845493803047223158 [ 7080.110571] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7080.457374] Lustre: server umount lustre-MDT0001 complete [ 7083.492208] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 16:32:39 (1786825959) [ 7097.791634] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 7107.168732] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 7141.279293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7141.581503] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7141.607813] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7141.682913] Lustre: lustre-MDT0000: new disk, initializing [ 7141.764239] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7146.083897] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7155.970801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7156.056727] Lustre: 218273:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7156.353529] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7160.489090] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7168.921757] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7173.458346] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7179.685311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7179.924509] Lustre: 220243:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7181.892460] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7185.769318] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7194.863942] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7199.739892] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7199.975919] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7203.886042] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7204.089091] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7207.401367] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7207.404750] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7207.407944] Lustre: Skipped 2 previous similar messages [ 7207.411574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7207.433456] Lustre: Skipped 11 previous similar messages [ 7220.191266] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7220.383131] Lustre: server umount lustre-OST0000 complete [ 7227.875280] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7230.169153] Lustre: server umount lustre-MDT0000 complete [ 7232.992254] LustreError: 219150:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7233.008989] LustreError: 219150:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7233.351797] LustreError: 218264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786826111 with bad export cookie 3845493803047224040 [ 7233.362640] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7233.369436] LustreError: 218264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 7233.654570] Lustre: server umount lustre-MDT0001 complete [ 7241.179098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7245.759849] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7253.380527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7253.666624] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7253.682572] Lustre: Skipped 5 previous similar messages [ 7258.553930] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7265.615403] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7271.055360] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7277.346723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7283.566283] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7290.272900] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7299.553586] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7299.559555] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7299.567273] Lustre: Skipped 3 previous similar messages [ 7299.571124] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7299.575565] Lustre: Skipped 6 previous similar messages [ 7305.810505] Lustre: server umount lustre-OST0000 complete [ 7323.106619] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7323.274709] Lustre: server umount lustre-MDT0000 complete [ 7327.203941] LustreError: 222439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786826205 with bad export cookie 3845493803047224922 [ 7327.207609] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7327.214552] LustreError: 222439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7327.707946] Lustre: server umount lustre-MDT0001 complete [ 7430.233806] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 7432.732648] Key type lgssc unregistered [ 7433.034951] LNet: 226125:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7433.055694] LNetError: 226125:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7434.092793] LNet: Removed LNI 192.168.206.157@tcp [ 7434.839121] Key type .llcrypt unregistered [ 7434.844145] Key type ._llcrypt unregistered [ 7458.130389] Key type ._llcrypt registered [ 7458.132378] Key type .llcrypt registered [ 7458.268816] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 7470.791750] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 7471.601808] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7471.663160] alg: No test for adler32 (adler32-zlib) [ 7472.737417] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [ 7472.989455] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 7474.687215] Key type lgssc registered [ 7476.008104] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7505.832921] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7505.848325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7507.048674] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7507.082544] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7507.141385] Lustre: lustre-MDT0000: new disk, initializing [ 7507.211206] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7507.226522] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7510.712735] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7519.646320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7519.789578] Lustre: 229455:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7519.908760] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7519.913758] Lustre: Skipped 1 previous similar message [ 7520.006227] Lustre: lustre-MDT0001: new disk, initializing [ 7520.103745] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7520.147650] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7520.165176] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7524.251338] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7532.560598] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7537.087824] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7542.400769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7542.635565] Lustre: lustre-OST0000: new disk, initializing [ 7542.639995] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7542.645931] Lustre: 231426:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7542.746492] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7547.852556] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7547.943772] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7547.952236] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7548.026479] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7556.706578] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7561.708541] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7561.877426] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7566.758235] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7566.937470] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7568.865494] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7568.882449] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7568.919632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7573.474876] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7573.479049] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7573.487399] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7574.878341] Lustre: server umount lustre-OST0000 complete [ 7578.594189] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7578.617384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7578.622620] Lustre: Skipped 2 previous similar messages [ 7583.716730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7583.721285] Lustre: Skipped 1 previous similar message [ 7584.517304] Lustre: server umount lustre-MDT0000 complete [ 7587.949297] LustreError: 229448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786826465 with bad export cookie 2172642668851038654 [ 7587.954187] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7587.964085] LustreError: 229448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7588.267700] Lustre: server umount lustre-MDT0001 complete [ 7604.576539] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 7613.530373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7613.868613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7617.654742] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7619.042746] LustreError: 234360:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7619.063207] LustreError: 234360:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7624.163438] LustreError: 234361:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7624.724709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7624.961264] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7629.126662] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7631.889842] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7638.454721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7638.831465] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7645.032537] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7652.485347] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7652.644343] Lustre: lustre-OST0001: new disk, initializing [ 7652.647696] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7652.653984] Lustre: 236667:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7654.674436] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7654.682860] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7654.782710] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7658.068410] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7668.086813] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7673.183246] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7682.900735] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 16:42:39 (1786826559) [ 7686.624792] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7686.632466] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7686.642381] Lustre: Skipped 1 previous similar message [ 7686.653354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7690.903348] Lustre: server umount lustre-MDT0000 complete [ 7693.785556] LustreError: 234342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786826571 with bad export cookie 2172642668851039536 [ 7693.790927] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7693.794141] LustreError: 234342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7694.044556] Lustre: server umount lustre-MDT0001 complete [ 7706.366804] Lustre: server umount lustre-OST0000 complete [ 7719.894670] Lustre: server umount lustre-OST0001 complete [ 7726.300658] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 7735.073858] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 7765.201855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7765.495704] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7765.528565] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7765.583756] Lustre: lustre-MDT0000: new disk, initializing [ 7765.635234] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7765.641493] Lustre: Skipped 1 previous similar message [ 7765.651670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7769.319052] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7779.958920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7780.115771] Lustre: 240610:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7780.211714] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7780.225971] Lustre: Skipped 1 previous similar message [ 7780.358553] Lustre: lustre-MDT0001: new disk, initializing [ 7780.576869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7780.583092] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7785.510377] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7795.024644] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7801.920285] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7809.175962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7809.502625] Lustre: lustre-OST0000: new disk, initializing [ 7809.507728] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7809.521209] Lustre: 242579:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7809.582051] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7809.585393] Lustre: Skipped 1 previous similar message [ 7810.948126] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7810.962069] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7811.176496] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7816.172110] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7825.469185] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7831.227509] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7831.484865] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7835.180868] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7835.327618] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7841.767763] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7841.771886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7841.772077] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7841.779609] Lustre: Skipped 4 previous similar messages [ 7841.789385] Lustre: Skipped 4 previous similar messages [ 7843.107669] Lustre: server umount lustre-OST0000 complete [ 7852.892505] Lustre: server umount lustre-MDT0000 complete [ 7855.788138] LustreError: 242578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786826733 with bad export cookie 2172642668851041671 [ 7855.790489] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7855.794690] LustreError: 242578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7855.988214] Lustre: server umount lustre-MDT0001 complete [ 7863.113569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7863.786117] LustreError: 244796:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7867.599200] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7868.897303] LustreError: 244797:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7874.018714] LustreError: 244796:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7874.913663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7875.210104] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7875.217670] Lustre: Skipped 1 previous similar message [ 7879.139225] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7885.136862] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7889.932665] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7895.184130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7900.610066] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7907.358848] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8052.203126] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8052.206763] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8052.226470] LustreError: Skipped 1 previous similar message [ 8052.235303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8052.238644] Lustre: Skipped 3 previous similar messages [ 8052.252426] Lustre: Skipped 6 previous similar messages [ 8057.880272] Lustre: server umount lustre-OST0000 complete [ 8075.231592] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8075.614039] Lustre: server umount lustre-MDT0000 complete [ 8077.793071] LustreError: 244797:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8077.806743] LustreError: 244797:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8078.860980] LustreError: 244776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786826956 with bad export cookie 2172642668851042553 [ 8078.866941] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8078.871966] LustreError: 244776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8079.120397] Lustre: server umount lustre-MDT0001 complete [ 8176.298515] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 8178.352390] Key type lgssc unregistered [ 8178.560981] LNet: 248405:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8178.570945] LNetError: 248405:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8178.583499] LNet: Removed LNI 192.168.206.157@tcp [ 8179.360116] Key type .llcrypt unregistered [ 8179.362587] Key type ._llcrypt unregistered [ 8198.073142] Key type ._llcrypt registered [ 8198.076832] Key type .llcrypt registered [ 8198.181684] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 8213.123117] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 8214.032800] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8214.110238] alg: No test for adler32 (adler32-zlib) [ 8215.394076] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [ 8215.671566] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 8217.383122] Key type lgssc registered [ 8219.089757] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8261.119672] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 8269.579090] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8269.613490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8270.876228] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8270.932922] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8271.055527] Lustre: lustre-MDT0000: new disk, initializing [ 8271.201484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8271.225840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8274.447845] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 8283.034417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8283.153822] Lustre: 252474:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8283.225916] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8283.237771] Lustre: Skipped 1 previous similar message [ 8283.313394] Lustre: lustre-MDT0001: new disk, initializing [ 8283.398732] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8283.434519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8283.446259] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8286.851379] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 8290.707817] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8296.577973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8296.882609] Lustre: lustre-OST0000: new disk, initializing [ 8296.891150] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8296.901624] Lustre: 254107:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8297.090077] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8302.149227] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 8302.684722] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8302.710450] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8302.828047] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8311.563342] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8311.769881] Lustre: lustre-OST0001: new disk, initializing [ 8311.772466] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8311.776579] Lustre: 254979:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8311.841825] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8317.195658] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 8318.535063] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8318.544982] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8318.612075] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8326.194174] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8332.537796] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8344.548096] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8344.552063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8344.585762] Lustre: Skipped 3 previous similar messages [ 8344.593828] Lustre: Skipped 3 previous similar messages [ 8347.870443] Lustre: server umount lustre-MDT0000 complete [ 8351.549172] LustreError: 254978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786827229 with bad export cookie 6767427681431767065 [ 8351.553922] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8351.562971] LustreError: 254978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8351.965569] Lustre: server umount lustre-MDT0001 complete [ 8366.124267] Lustre: server umount lustre-OST0000 complete [ 8378.677309] Lustre: server umount lustre-OST0001 complete [ 8390.299084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8397.994459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8405.052712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 8413.372878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 8429.345423] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 8439.053442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8439.073305] Lustre: lustre-MDT0000: reset Object Index mappings [ 8439.427893] LustreError: 259984:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8439.436645] LustreError: 259984:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 8439.502928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8443.473720] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 8444.897940] LustreError: 259985:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8450.019880] LustreError: 259984:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8451.403373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8451.433402] Lustre: lustre-MDT0001: reset Object Index mappings [ 8451.876705] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8455.450591] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 8458.004824] Lustre: 261125:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8464.365750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8470.130297] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 8476.972854] LustreError: 261477:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8477.625876] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8477.896261] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8477.899177] Lustre: Skipped 1 previous similar message [ 8483.527799] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 8484.362509] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8484.362592] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 8490.206368] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8493.354249] Lustre: 262996:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8498.146512] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8498.151668] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8498.165986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8499.681979] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8499.690299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8499.699949] Lustre: Skipped 1 previous similar message [ 8499.704098] Lustre: Skipped 1 previous similar message [ 8503.327562] Lustre: server umount lustre-MDT0000 complete [ 8504.807652] LustreError: 259984:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8504.828534] LustreError: 259984:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8506.593173] LustreError: 262308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786827384 with bad export cookie 6767427681431773855 [ 8506.594031] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8506.599956] LustreError: 262308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8506.844869] Lustre: server umount lustre-MDT0001 complete [ 8520.949054] Lustre: server umount lustre-OST0000 complete [ 8535.593825] Lustre: server umount lustre-OST0001 complete [ 8553.672644] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 16:57:09 (1786827429) [ 8567.930547] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 8578.255737] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 8610.893514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8611.095666] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8611.122746] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8611.183219] Lustre: lustre-MDT0000: new disk, initializing [ 8611.247288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8611.259517] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8615.137371] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 8624.961482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8625.177951] Lustre: 267250:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8625.323157] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8625.330256] Lustre: Skipped 1 previous similar message [ 8625.533199] Lustre: lustre-MDT0001: new disk, initializing [ 8625.699121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8625.715041] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8632.271891] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 8643.091837] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8649.393755] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8656.218360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8656.608433] Lustre: lustre-OST0000: new disk, initializing [ 8656.612407] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8656.626699] Lustre: 269218:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8658.208481] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8658.232213] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8658.397248] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8663.764934] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 8672.114575] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8676.370105] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8676.612291] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8681.135414] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8681.440079] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8684.020473] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8684.024331] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8684.027332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8684.027348] Lustre: Skipped 2 previous similar messages [ 8684.053576] Lustre: Skipped 2 previous similar messages [ 8689.121873] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8689.139612] Lustre: Skipped 2 previous similar messages [ 8689.645586] Lustre: server umount lustre-OST0000 complete [ 8694.240574] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8699.360586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8699.366203] Lustre: Skipped 3 previous similar messages [ 8700.484645] Lustre: server umount lustre-MDT0000 complete [ 8704.485118] LustreError: 267260:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8704.515630] LustreError: 267260:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8704.560257] LustreError: 267242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786827582 with bad export cookie 6767427681431775948 [ 8704.571590] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8704.575367] LustreError: 267242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8704.904227] Lustre: server umount lustre-MDT0001 complete [ 8722.635725] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 8732.017764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8732.375089] LustreError: 272152:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8732.416964] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8732.419738] Lustre: Skipped 2 previous similar messages [ 8736.715362] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 8743.397173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8748.217214] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 8751.399510] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8757.740989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8765.386315] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 8772.627635] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8772.725586] Lustre: lustre-OST0001: new disk, initializing [ 8772.728744] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8772.734217] Lustre: 274457:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8774.194844] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8774.202683] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8774.365288] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8778.785341] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 8787.210775] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8790.050621] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8794.418670] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 8794.592789] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 8803.297466] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8803.300863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8803.316224] Lustre: Skipped 4 previous similar messages [ 8803.323208] Lustre: Skipped 3 previous similar messages [ 8807.045218] Lustre: server umount lustre-MDT0000 complete [ 8808.422422] LustreError: 272152:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8808.446323] LustreError: 272152:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 8810.834339] LustreError: 272132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786827688 with bad export cookie 6767427681431776830 [ 8810.843159] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8810.849547] LustreError: 272132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8811.199612] Lustre: server umount lustre-MDT0001 complete [ 8824.984678] Lustre: server umount lustre-OST0000 complete [ 8837.579396] Lustre: server umount lustre-OST0001 complete [ 8850.896126] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 8859.108093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8863.729623] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 8870.588492] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8870.885758] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8870.895814] Lustre: Skipped 4 previous similar messages [ 8874.712817] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 8877.166876] Lustre: 278672:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8883.386848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8884.789078] LustreError: 279026:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8884.821565] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 8884.826539] LustreError: 279026:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8889.727326] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 8896.937089] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8902.517792] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 8902.647881] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 8908.748849] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8911.874665] Lustre: 280545:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8916.227757] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 8916.263870] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 8924.031847] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 17:03:20 (1786827800) [ 8938.468124] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8938.476596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8938.482429] Lustre: Skipped 3 previous similar messages [ 8938.495760] Lustre: Skipped 3 previous similar messages [ 8943.938876] Lustre: server umount lustre-MDT0000 complete [ 8946.594082] LustreError: 278273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786827824 with bad export cookie 6767427681431783368 [ 8946.596660] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8946.603571] LustreError: 278273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8953.099776] Lustre: server umount lustre-MDT0001 complete [ 8962.604838] Lustre: server umount lustre-OST0000 complete [ 8972.176527] Lustre: server umount lustre-OST0001 complete [ 8978.193159] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 8987.827363] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 9032.662279] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 9042.541043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9042.753661] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9042.774325] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9042.842488] Lustre: lustre-MDT0000: new disk, initializing [ 9042.913134] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9047.411908] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 9056.939664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9057.072414] Lustre: 284786:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9057.281228] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9061.729617] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 9074.888754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9075.302685] Lustre: 286418:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9077.366888] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9082.291701] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 9091.796961] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9091.972516] Lustre: 287288:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9093.942277] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9093.950589] Lustre: Skipped 1 previous similar message [ 9094.075573] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9098.260246] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 9107.567261] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9111.701710] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9111.721225] Lustre: Skipped 1 previous similar message [ 9134.422564] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 17:06:50 (1786828010) [ 9139.168423] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9139.176493] LustreError: Skipped 1 previous similar message [ 9139.180294] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9139.189236] Lustre: Skipped 2 previous similar messages [ 9139.195364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9139.202990] Lustre: Skipped 8 previous similar messages [ 9142.300296] Lustre: server umount lustre-MDT0000 complete [ 9145.314400] LustreError: 284798:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9145.343155] LustreError: 284798:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 9145.406096] LustreError: 284777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786828023 with bad export cookie 6767427681431791012 [ 9145.413906] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9145.418871] LustreError: 284777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9145.641788] Lustre: server umount lustre-MDT0001 complete [ 9158.741208] Lustre: server umount lustre-OST0000 complete [ 9174.063652] Lustre: server umount lustre-OST0001 complete [ 9182.170716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9182.768774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9182.775885] Lustre: Skipped 6 previous similar messages [ 9186.845799] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 9193.973582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9198.050727] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 9203.733787] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9208.061988] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9213.309866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9219.086623] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 9226.316268] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9246.132197] loop1: detected capacity change from 0 to 204800000 [ 9246.522648] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9246.538197] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9246.556494] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9250.800717] loop1: detected capacity change from 0 to 204800000 [ 9255.763920] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9255.835660] LustreError: 289527:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) fs12454: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 9255.852306] LustreError: 289527:0:(mgs_llog.c:731:mgs_check_index()) Can't get db for fs12454 [ 9255.856217] LustreError: 289527:0:(mgs_handler.c:517:mgs_target_reg()) fs12454-MDT0001: Error checking target index: rc = -13 [ 9255.866991] LustreError: fs12454-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 9255.872665] LustreError: 293257:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 9255.876390] LustreError: 293257:0:(tgt_mount.c:2129:server_put_super()) no obd fs12454-MDT0001 [ 9255.879174] LustreError: 293257:0:(tgt_mount.c:116:server_deregister_mount()) fs12454-MDT0001 not registered [ 9255.906271] Lustre: server umount fs12454-MDT0001 complete [ 9255.907814] LustreError: 293257:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 9262.410344] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9262.489791] Lustre: Setting parameter fs12454-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs12454-MDT0001 [ 9262.525438] Lustre: srv-fs12454-MDT0001: No data found on store. Initialize space. [ 9262.529301] Lustre: Skipped 4 previous similar messages [ 9262.583290] Lustre: fs12454-MDT0001: new disk, initializing [ 9262.585730] Lustre: Skipped 3 previous similar messages [ 9262.685141] Lustre: srv-fs12454-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 9262.705134] Lustre: 293724:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs12454-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9262.712921] LustreError: 293724:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs12454-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9266.837550] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 9273.936889] Lustre: server umount fs12454-MDT0001 complete [ 9286.786508] loop1: detected capacity change from 0 to 204800000 [ 9291.755583] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9292.287294] Lustre: 295293:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs12454-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9292.311835] LustreError: 295293:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs12454-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9295.906630] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 9302.466478] Lustre: server umount fs12454-MDT0001 complete [ 9304.054101] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [ 9309.187851] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [ 9313.964301] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.57@tcp (stopping) [ 9313.970968] Lustre: Skipped 4 previous similar messages [ 9314.279522] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9314.291556] Lustre: Skipped 5 previous similar messages [ 9318.644187] Lustre: server umount lustre-MDT0000 complete [ 9325.316475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9325.379293] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9330.227656] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 9330.677464] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9340.161755] loop1: detected capacity change from 0 to 204800000 [ 9346.058145] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9346.556084] Lustre: 297739:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs12454-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9346.572592] LustreError: 297739:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs12454-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9351.330607] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 9356.033891] Lustre: server umount fs12454-MDT0001 complete [ 9376.742045] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9380.765653] Lustre: server umount lustre-OST0000 complete [ 9398.240649] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9398.384566] Lustre: server umount lustre-MDT0000 complete [ 9401.677042] LustreError: 294270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786828279 with bad export cookie 6767427681431793693 [ 9401.677867] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9401.684339] LustreError: 294270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9402.161627] Lustre: server umount lustre-MDT0001 complete [ 9412.127871] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 9414.823830] Key type lgssc unregistered [ 9415.146599] LNet: 300084:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9415.155210] LNetError: 300084:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9415.190830] LNet: Removed LNI 192.168.206.157@tcp [ 9416.102167] Key type .llcrypt unregistered [ 9416.105360] Key type ._llcrypt unregistered [ 9437.112719] Key type ._llcrypt registered [ 9437.114731] Key type .llcrypt registered [ 9437.194372] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 9437.897433] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9437.937089] alg: No test for adler32 (adler32-zlib) [ 9438.885466] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [ 9439.026722] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 9440.647210] Key type lgssc registered [ 9441.762074] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9449.327836] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9449.348731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9450.817725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9454.529302] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 9461.217376] LustreError: 301355:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9461.235087] LustreError: 301355:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9461.571470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9461.995044] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9465.884485] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 9472.265399] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9477.058192] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9482.593329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9482.959792] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9487.110058] LustreError: 303185:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9488.677536] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 9492.453925] LustreError: 303185:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9495.443329] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9496.550347] LustreError: 303184:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9496.573205] LustreError: 303184:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9501.676143] LustreError: 303185:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9501.705976] LustreError: 303185:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9511.289677] loop1: detected capacity change from 0 to 204800000 [ 9511.496718] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9511.509676] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9511.517097] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9511.926228] LustreError: 303455:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9511.946969] LustreError: 303455:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 9515.965673] loop1: detected capacity change from 0 to 204800000 [ 9521.075166] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9521.266830] Lustre: fs8690-OST0000: new disk, initializing [ 9521.277660] Lustre: srv-fs8690-OST0000: No data found on store. Initialize space. [ 9521.292143] Lustre: 305088:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9521.370236] Lustre: fs8690-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9527.173859] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 9528.966331] LustreError: 303856:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9528.995848] LustreError: 303856:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 9534.524258] Lustre: server umount fs8690-OST0000 complete [ 9544.914642] loop1: detected capacity change from 0 to 204800000 [ 9549.889073] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9550.113848] Lustre: fs8690-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9555.160660] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 9562.060709] loop2: detected capacity change from 0 to 204800000 [ 9562.264463] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9562.301557] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9562.315733] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9563.110899] LustreError: 303856:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9563.132868] LustreError: 303856:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 19 previous similar messages [ 9566.305942] loop2: detected capacity change from 0 to 204800000 [ 9570.539797] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9570.626855] LustreError: fs8690-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 9570.634147] LustreError: fs8690-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 9570.640897] LustreError: 308125:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 9570.649491] LustreError: 308125:0:(tgt_mount.c:2129:server_put_super()) no obd fs8690-OST0001 [ 9570.652619] LustreError: 308125:0:(tgt_mount.c:116:server_deregister_mount()) fs8690-OST0001 not registered [ 9570.676740] Lustre: server umount fs8690-OST0001 complete [ 9570.679456] LustreError: 308125:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 9573.355590] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [ 9578.466596] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [ 9579.298657] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9579.403083] Lustre: fs8690-OST0001: new disk, initializing [ 9579.406188] Lustre: srv-fs8690-OST0001: No data found on store. Initialize space. [ 9579.410662] Lustre: 308677:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9579.501643] Lustre: fs8690-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9585.040426] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 9592.572438] Lustre: server umount fs8690-OST0000 complete [ 9600.133835] Lustre: server umount fs8690-OST0001 complete [ 9609.314432] loop1: detected capacity change from 0 to 204800000 [ 9613.773827] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9614.072040] Lustre: fs8690-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9619.619771] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 9626.009636] loop2: detected capacity change from 0 to 204800000 [ 9629.666730] LustreError: 303455:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9629.683762] LustreError: 303455:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 38 previous similar messages [ 9630.809661] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9636.173523] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 9644.296561] Lustre: server umount fs8690-OST0000 complete [ 9652.898129] Lustre: server umount fs8690-OST0001 complete [ 9665.507646] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9665.514546] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9665.516570] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9665.525798] Lustre: Skipped 1 previous similar message [ 9669.537156] Lustre: server umount lustre-OST0000 complete [ 9675.747066] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9675.751667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9675.760700] Lustre: Skipped 1 previous similar message [ 9675.774882] Lustre: Skipped 2 previous similar messages [ 9679.332268] Lustre: server umount lustre-MDT0000 complete [ 9682.379500] LustreError: 301336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786828560 with bad export cookie 17537935848571525659 [ 9682.388085] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9682.388224] LustreError: 301336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9682.667259] Lustre: server umount lustre-MDT0001 complete [ 9691.448571] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 9693.736152] Key type lgssc unregistered [ 9693.986246] LNet: 314595:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9693.992735] LNetError: 314595:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9694.001718] LNet: Removed LNI 192.168.206.157@tcp [ 9694.728303] Key type .llcrypt unregistered [ 9694.733971] Key type ._llcrypt unregistered [ 9719.361181] Key type ._llcrypt registered [ 9719.367814] Key type .llcrypt registered [ 9719.463654] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 9731.629039] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 9732.527215] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9732.600946] alg: No test for adler32 (adler32-zlib) [ 9733.662754] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [ 9733.914611] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 9735.599141] Key type lgssc registered [ 9736.661829] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9770.826296] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9770.857838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9772.204892] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9772.228569] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9772.323734] Lustre: lustre-MDT0000: new disk, initializing [ 9772.414488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9772.429097] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9776.636577] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 9784.933625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9785.049436] Lustre: 318434:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9785.102685] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9785.106789] Lustre: Skipped 1 previous similar message [ 9785.201488] Lustre: lustre-MDT0001: new disk, initializing [ 9785.257205] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9785.289829] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9785.302459] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9788.953434] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 9797.394297] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9802.078305] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9808.637419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9808.800368] Lustre: lustre-OST0000: new disk, initializing [ 9808.803557] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9808.808405] Lustre: 320403:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9808.875119] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9815.266278] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 9818.731406] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9818.744555] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9818.882632] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9824.028934] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9826.409251] Lustre: Mounted lustre-client [ 9827.573161] Lustre: Unmounted lustre-client [ 9836.517277] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9836.522066] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9836.535769] Lustre: Skipped 1 previous similar message [ 9841.638795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9841.645828] Lustre: Skipped 2 previous similar messages [ 9841.863722] Lustre: server umount lustre-OST0000 complete [ 9846.757314] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9846.759261] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9846.763128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9846.786748] Lustre: Skipped 1 previous similar message [ 9851.279337] Lustre: server umount lustre-MDT0000 complete [ 9854.846684] LustreError: 318429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786828732 with bad export cookie 1416147579471932311 [ 9854.853181] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9854.858689] LustreError: 318429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9855.448777] Lustre: server umount lustre-MDT0001 complete [ 9864.390830] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 9866.730298] Key type lgssc unregistered [ 9877.215512] LNet: 322566:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9877.226714] LNetError: 322566:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9959.143081] LNet: Removed LNI 192.168.206.157@tcp [ 9959.966893] Key type .llcrypt unregistered [ 9959.969562] Key type ._llcrypt unregistered [ 9975.224877] Key type ._llcrypt registered [ 9975.228314] Key type .llcrypt registered [ 9975.314766] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 9990.209166] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 9991.088706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9991.142927] alg: No test for adler32 (adler32-zlib) [ 9992.268911] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [ 9992.475551] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 9994.111242] Key type lgssc registered [ 9995.718959] Lustre: Echo OBD driver; http://www.lustre.org/ [10025.495690] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 17:21:41 (1786828901) [10031.615456] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10031.637985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10032.916100] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10032.958835] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10033.067783] Lustre: lustre-MDT0000: new disk, initializing [10033.151044] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10033.170546] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10037.679879] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10046.591827] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10046.714356] Lustre: 325788:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10046.754607] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10046.757992] Lustre: Skipped 1 previous similar message [10046.822787] Lustre: lustre-MDT0001: new disk, initializing [10046.908736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10046.946137] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10046.961896] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10050.898304] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10059.189974] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10064.041771] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10070.323566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10070.618224] Lustre: lustre-OST0000: new disk, initializing [10070.621933] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10070.627599] Lustre: 327755:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10070.706215] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10077.024962] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10080.323832] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10080.330829] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10080.413436] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10086.363526] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10106.340454] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10106.342196] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10106.344624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10106.353539] Lustre: Skipped 1 previous similar message [10107.910278] Lustre: server umount lustre-OST0000 complete [10111.457121] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10111.458115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10111.462958] Lustre: Skipped 1 previous similar message [10111.469960] Lustre: Skipped 1 previous similar message [10116.578040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10116.581124] Lustre: Skipped 1 previous similar message [10117.403705] Lustre: server umount lustre-MDT0000 complete [10121.260671] LustreError: 325782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786828999 with bad export cookie 12807785626954769556 [10121.281255] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10121.291028] LustreError: 325782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10122.215741] Lustre: server umount lustre-MDT0001 complete [10133.306746] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [10136.021114] Key type lgssc unregistered [10136.386653] LNet: 329663:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10136.394219] LNetError: 329663:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10137.447887] LNet: Removed LNI 192.168.206.157@tcp [10138.235327] Key type .llcrypt unregistered [10138.238563] Key type ._llcrypt unregistered [10156.423620] Key type ._llcrypt registered [10156.430695] Key type .llcrypt registered [10156.595658] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [10157.808901] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10157.861900] alg: No test for adler32 (adler32-zlib) [10159.083174] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [10159.348302] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [10161.095255] Key type lgssc registered [10162.134151] Lustre: Echo OBD driver; http://www.lustre.org/ [10170.360990] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10170.383430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10171.822492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10175.998387] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10182.112923] LustreError: 330943:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10182.138527] LustreError: 330943:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10183.886973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10184.190615] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10188.095076] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10194.616548] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10200.215917] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10206.544414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10206.845897] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10213.027187] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10220.299737] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10228.333483] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10229.595329] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10232.808078] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10232.809868] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10232.831170] Lustre: Skipped 2 previous similar messages [10232.840962] Lustre: Skipped 2 previous similar messages [10237.605258] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.57@tcp (stopping) [10238.929918] Lustre: server umount lustre-MDT0000 complete [10242.242842] LustreError: 330922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786829120 with bad export cookie 9742232609183789118 [10242.244180] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10242.259425] LustreError: 330922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10242.693671] LustreError: 330939:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10242.721513] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.57@tcp (stopping) [10242.726556] Lustre: Skipped 3 previous similar messages [10243.046624] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10247.073158] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10247.073178] LustreError: 330939:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10247.083673] Lustre: Skipped 1 previous similar message [10247.102934] LustreError: 330939:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10248.759102] Lustre: server umount lustre-MDT0001 complete [10252.784638] Lustre: server umount lustre-OST0000 complete [10261.005211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10265.008720] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10267.325210] Lustre: 335122:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.157@tcp [10269.110901] Lustre: server umount MGS complete [10276.645588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10277.103870] LustreError: 335694:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10277.125837] LustreError: 335694:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10277.176214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10281.059734] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10282.469197] LustreError: 335695:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10290.017309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10290.448869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10294.726623] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10302.576448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10302.969433] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10308.623773] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10315.557847] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10316.982585] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [10321.756719] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10325.576816] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [10331.621803] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10331.626095] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10331.635856] LustreError: Skipped 1 previous similar message [10331.637776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10331.649722] Lustre: Skipped 1 previous similar message [10331.688269] Lustre: Skipped 2 previous similar messages [10335.580201] Lustre: server umount lustre-OST0000 complete [10341.857254] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10341.858422] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10341.864547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10341.900438] Lustre: Skipped 1 previous similar message [10344.961622] Lustre: server umount lustre-MDT0000 complete [10346.990209] LustreError: 335694:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10347.020084] LustreError: 335694:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10348.610325] LustreError: 335676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786829226 with bad export cookie 9742232609183789958 [10348.618136] LustreError: 335676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10348.618472] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10348.932302] Lustre: server umount lustre-MDT0001 complete [10358.935874] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [10361.815858] Key type lgssc unregistered [10362.095201] LNet: 339130:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10362.105872] LNetError: 339130:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10362.132284] LNet: Removed LNI 192.168.206.157@tcp [10362.913553] Key type .llcrypt unregistered [10362.920562] Key type ._llcrypt unregistered [10380.370791] Key type ._llcrypt registered [10380.373916] Key type .llcrypt registered [10380.493265] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [10395.401349] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [10396.724853] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10396.749924] alg: No test for adler32 (adler32-zlib) [10397.885360] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [10398.140325] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [10399.783672] Key type lgssc registered [10401.123526] Lustre: Echo OBD driver; http://www.lustre.org/ [10432.048261] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10432.061930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10433.280257] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10433.311938] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10433.432888] Lustre: lustre-MDT0000: new disk, initializing [10433.591721] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10433.619217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10438.060553] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10448.476329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10448.684711] Lustre: 342290:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10448.767595] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10448.772146] Lustre: Skipped 1 previous similar message [10448.849831] Lustre: lustre-MDT0001: new disk, initializing [10448.935633] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10448.975172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10448.979689] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10453.750384] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10462.746405] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10468.412958] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10474.669334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10474.912456] Lustre: lustre-OST0000: new disk, initializing [10474.918861] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10474.928244] Lustre: 344258:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10475.008860] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10480.478251] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10481.179419] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10481.190031] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10481.300057] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10488.508907] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10492.782129] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10492.987574] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10497.136967] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10497.361675] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10501.603252] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10501.612577] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10501.623110] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10502.117271] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10502.124466] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10502.132501] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10502.135054] Lustre: Skipped 1 previous similar message [10507.232978] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10507.244849] Lustre: Skipped 1 previous similar message [10512.357042] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10512.369101] Lustre: Skipped 1 previous similar message [10513.375903] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10513.519427] Lustre: server umount lustre-OST0000 complete [10517.477061] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10517.494188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10517.504090] Lustre: Skipped 1 previous similar message [10527.714739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10527.726076] Lustre: Skipped 3 previous similar messages [10531.295109] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10531.590913] Lustre: server umount lustre-MDT0000 complete [10535.738541] LustreError: 342280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786829413 with bad export cookie 10958048582087492124 [10535.768403] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10536.106339] Lustre: server umount lustre-MDT0001 complete [10545.786392] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 17:30:21 (1786829421) [10558.394262] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [10567.008870] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [10597.442617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10597.665466] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10597.699257] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10597.763415] Lustre: lustre-MDT0000: new disk, initializing [10597.806293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10597.818174] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10601.933753] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10611.342856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10611.453888] Lustre: 349015:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10611.489050] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10611.492052] Lustre: Skipped 1 previous similar message [10611.548243] Lustre: lustre-MDT0001: new disk, initializing [10611.624036] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10611.652104] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10611.669045] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10616.130375] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10624.094116] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10628.825594] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10634.394487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10634.775877] Lustre: lustre-OST0000: new disk, initializing [10634.779745] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10634.788379] Lustre: 350985:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10634.862074] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10641.286263] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10643.518449] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10643.529418] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10643.618796] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10650.139408] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10679.781166] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10679.782700] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10679.789512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10679.798323] Lustre: Skipped 2 previous similar messages [10679.810187] Lustre: Skipped 1 previous similar message [10684.916147] Lustre: server umount lustre-OST0000 complete [10688.480420] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10688.487525] LustreError: Skipped 1 previous similar message [10688.489363] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10702.304570] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10702.518714] Lustre: server umount lustre-MDT0000 complete [10704.352567] LustreError: 349028:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10704.366149] LustreError: 349028:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10705.639217] LustreError: 349009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786829583 with bad export cookie 10958048582087493006 [10705.647620] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10705.654698] LustreError: 349009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [10705.961481] Lustre: server umount lustre-MDT0001 complete [10715.003577] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [10717.658724] Key type lgssc unregistered [10717.995741] LNet: 353137:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10718.002624] LNetError: 353137:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10718.012790] LNet: Removed LNI 192.168.206.157@tcp [10718.832087] Key type .llcrypt unregistered [10718.836400] Key type ._llcrypt unregistered [10800.959747] Key type ._llcrypt registered [10800.961436] Key type .llcrypt registered [10801.047937] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [10801.304536] Key type .llcrypt unregistered [10801.306568] Key type ._llcrypt unregistered [10876.166185] Key type ._llcrypt registered [10876.168756] Key type .llcrypt registered [10876.347558] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [10876.761298] Key type .llcrypt unregistered [10876.763972] Key type ._llcrypt unregistered [10904.677024] Key type ._llcrypt registered [10904.678852] Key type .llcrypt registered [10904.784619] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [10904.981358] Key type .llcrypt unregistered [10904.986295] Key type ._llcrypt unregistered [10931.029287] Key type ._llcrypt registered [10931.034362] Key type .llcrypt registered [10931.141558] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [10931.380412] Key type .llcrypt unregistered [10931.384643] Key type ._llcrypt unregistered [10995.740992] Key type ._llcrypt registered [10995.742972] Key type .llcrypt registered [10995.861675] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [10996.128255] Key type .llcrypt unregistered [10996.130822] Key type ._llcrypt unregistered [11020.483570] Key type ._llcrypt registered [11020.485417] Key type .llcrypt registered [11020.571430] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [11020.782401] Key type .llcrypt unregistered [11020.791395] Key type ._llcrypt unregistered [11043.540151] Key type ._llcrypt registered [11043.543741] Key type .llcrypt registered [11043.594339] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [11043.802274] Key type .llcrypt unregistered [11043.804048] Key type ._llcrypt unregistered [11069.863775] Key type ._llcrypt registered [11069.865877] Key type .llcrypt registered [11070.017657] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [11070.256503] Key type .llcrypt unregistered [11070.260141] Key type ._llcrypt unregistered [11100.875628] Key type ._llcrypt registered [11100.877537] Key type .llcrypt registered [11101.017331] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [11101.354352] Key type .llcrypt unregistered [11101.355937] Key type ._llcrypt unregistered [11129.093645] Key type ._llcrypt registered [11129.097712] Key type .llcrypt registered [11129.209917] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [11129.485278] Key type .llcrypt unregistered [11129.491193] Key type ._llcrypt unregistered [11185.939146] Key type ._llcrypt registered [11185.944960] Key type .llcrypt registered [11186.082173] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [11186.262428] Key type .llcrypt unregistered [11186.264451] Key type ._llcrypt unregistered [11213.095212] Key type ._llcrypt registered [11213.104118] Key type .llcrypt registered [11213.272916] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [11213.569306] Key type .llcrypt unregistered [11213.571643] Key type ._llcrypt unregistered [11240.528246] Key type ._llcrypt registered [11240.529993] Key type .llcrypt registered [11240.618825] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [11240.895491] Key type .llcrypt unregistered [11240.899635] Key type ._llcrypt unregistered [11285.388185] Key type ._llcrypt registered [11285.393183] Key type .llcrypt registered [11285.480553] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [11297.672456] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [11298.614795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11298.640130] alg: No test for adler32 (adler32-zlib) [11299.697000] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [11300.017132] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [11301.663115] Key type lgssc registered [11302.554154] Lustre: Echo OBD driver; http://www.lustre.org/ [11332.724443] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11332.766136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11334.030918] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [11334.066965] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [11334.147370] Lustre: tst250fs-MDT0000: new disk, initializing [11334.252386] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11334.272210] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11338.727262] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [11347.913959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11348.030861] Lustre: 373308:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11348.093255] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [11348.097699] Lustre: Skipped 1 previous similar message [11348.174342] Lustre: tst250fs-MDT0001: new disk, initializing [11348.277348] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11348.310724] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11348.327026] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11352.584391] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [11356.916694] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11362.914939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11363.171447] Lustre: tst250fs-OST0000: new disk, initializing [11363.175089] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [11363.181244] Lustre: 374941:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11363.279665] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11367.982772] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11368.009138] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11368.144297] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11368.423468] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [11377.747984] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11377.929118] Lustre: tst250fs-OST0001: new disk, initializing [11377.932543] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [11377.937966] Lustre: 375811:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11378.070483] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11383.847802] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11383.865452] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11383.958608] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11384.181071] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [11389.089848] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [11391.406814] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [11391.412752] Lustre: Skipped 2 previous similar messages [11409.895709] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11409.900080] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11409.905345] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [11409.937637] Lustre: Skipped 3 previous similar messages [11415.009342] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [11415.019574] Lustre: Skipped 3 previous similar messages [11420.132347] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [11420.142097] Lustre: Skipped 5 previous similar messages [11422.176373] Lustre: tst250fs-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11422.526865] Lustre: server umount tst250fs-MDT0000 complete [11426.522285] LustreError: 373302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786830304 with bad export cookie 7014757179547565374 [11426.527771] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11426.536958] LustreError: 373302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11426.901848] Lustre: server umount tst250fs-MDT0001 complete [11440.436743] Lustre: server umount tst250fs-OST0000 complete [11454.149537] Lustre: server umount tst250fs-OST0001 complete [11470.513433] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [11478.754425] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [11507.544744] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 17:46:23 (1786830383) [11509.086373] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [11526.692466] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [11535.463155] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [11567.061664] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 11327 sec ========= 17:47:23 (1786830443) [11568.684438] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 17:47:24 (1786830444) === [11571.995976] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 17:47:27 (1786830447) === [11608.485863] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [11611.374673] Key type lgssc unregistered [11611.617235] LNet: 384956:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11611.625286] LNetError: 384956:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11611.646467] LNet: Removed LNI 192.168.206.157@tcp [11612.487240] Key type .llcrypt unregistered [11612.489955] Key type ._llcrypt unregistered